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FOREWORD 


Computer-based  training  in  the  Army  has  been  around  for  quite  some  time; 
however,  new  advances  in  computer  capabilities  will  allow  for  even  more 
sophisticated  human-computer  interaction  than  before.  One  example  is  interactive 
student-tutor  or  mixed  initiative  dialogues  in  computer  tutors.  While  the 
promise  is  present,  the  reality  of  such  an  interface  requires  further  investigation. 
Natural  language  processing  requirements  for  such  an  interface  and  the  accurate 
interpretation  of  student  questions  by  the  tutor  is  one  step  in  the  direction 
of  refining  such  a  capability.  The  results  of  this  work  will  provide  insight 
into  the  psychology  of  natural  language  processing  and  instructional  strategies 
useful  for  developing  improved  training. 

Army  trainers  and  developers  of  advanced  computer  tutors  will  ultimately 
profit  from  this  work  as  we  at  U.S.  Army  Research  Institute  begin  to  discover 
the  requirements  for  optimal  human-computer  interaction. 


Technical  Director 


i. 


Requirement : 

The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 
(ARI)  conducts  research  in  support  of  new  technologies  for  Army  training.  One 
particular  problem  is  that  of  effective  human-computer  interfaces.  This  report 
on  the  natural  language  processing  requirements  in  tutorial  dialogues  provides 
information  that  will  be  useful  for  developers  of  such  interfaces. 


Procedure : 

The  author  of  this  report  is  conducting  protocol  analysis  of  student¬ 
generated  questions  in  a  computer  tutor-like  learning  situation.  The  research 
has  developed  and  tested  an  interpretation  model  for  the  questions  to  allow  the 
tutor/teacher  to  describe  the  student  knowledge  state. 


Findings : 

A  specific  functional  question  set  appears  to  have  emerged  in  the  domain 
of  procedural  skill  learning.  A  tripartite  of  question  types  function  as 
information  gatherers  and  structural  organizers  for  the  domain  knowledge  during 
early  learning.  Articulated  questions  about  a  specific  problem  during  learning 
seem  to  activate  associated  knowledge  that  becomes  available  for  solving  and 
reasoning  about  the  problem. 


Utilization  of  Findings: 

The  purpose  of  this  report  is  to  describe  the  basic  cognitive  parameters 
involved  in  student  questions  asked  during  tutorial  dialogues.  These  results 
will  provide  a  basis  for  development  of  a  psychological  theory  of  question 
asking  as  a  metacognitive  process  during  learning.  Further,  this  work  will 
lead  to  understanding  the  natural  language  requirements  necessary  to  develop  a 
tutorial  dialogue  interface. 
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QUESTIONS  ASKED  DURING  COMMAND  LANGUAGE  LEARNING: 
IMPLICATIONS  FOR  KNOWLEDGE  REPRESENTATION 


INTRODUCTION 

Recent  advances  in  computer  technology  have  increased  the  need  for  people 
to  acquire  computer  skills  in  order  to  function  in  a  rapidly  developing 
computerized  society.  Users  interact  with  computers  for  a  variety  of  reasons: 
performing  tasks,  solving  problems,  and  learning  new  computer  skills.  Learning 
command  languages,  for  example,  those  used  for  word  processing,  database,  and 
spreadsheet  applications,  has  recently  become  required  in  automated  environ¬ 
ments.  Investigation  into  the  learning  process  that  occurs  when  one  acquires 
these  skills  can  serve  several  purposes  useful  for  understanding  human-computer 
interaction. 

Card,  Moran,  and  Newell  (1984)  propose  a  theoretical  framework,  GOMS 
(goals-operators-methods-selection) ,  for  representing  the  hierarchical  goal 
structure  for  complex  cognitive  skills.  According  to  this  theory,  the  cogni¬ 
tive  structure  for  the  skilled  behavior  consists  of  the  four  components  in  the 
model.  However,  the  GOMS  model  is  a  theoretical  construct  that  deals  only  with 
well-learned  skills.  Along  the  continuum  of  learning,  novices  rely  heavily  on 
general  problem  solving  behavior  for  performing  tasks  until  an  adequate 
database  and  structure  for  the  knowledge  is  acquired  (Anderson,  1982,  1985). 

The  transition  made  from  a  problem-solving  model  to  a  GOMS  model  during 
learning,  however,  is  poorly  understood.  Studies  investigating  the  change  in 
strategies  in  a  learning  by  doing  paradigm  (Anzai  &  Simon,  1979;  Card,  Moran, 

&  Newell,  1984)  have  shown  that  skilled  behavior  is  the  result  of  practice  and 
experience  with  the  task.  However,  more  information  is  needed  to  better  under¬ 
stand  this  process  and  how  the  knowledge  is  represented.  How  much  and  what 
type  of  domain  knowledge  is  needed  to  select  the  appropriate  method  to  accom¬ 
plish  a  task?  How  much  practice  is  needed  before  the  correct  selection  rules 
are  recognized  to  control  search  over  the  problem  space?  When  In  the  learning 
process  does  skilled  behavior  begin  to  emerge?  What  kind  of  instructional 
feedback  should  be  offered  and  when  to  optimize  learning?  The  answer  to 
these  questions  can  be  found  by  looking  at  the  nature  of  questions  themselves: 
questions  asked  during  skill  acquisition. 

During  learning,  students  will  ask  questions  to  obtain  the  information  they 
need  in  order  to  understand  the  task.  However,  there  currently  exists  no  psy¬ 
chological  theory  of  question  asking.  One  approach  toward  developing  a  theory 
of  question  asking  is  to  study  and  classify  the  types  of  questions  and  their 
function  within  a  particular  system.  Harrah  (1973)  points  out  that  one  learns 
the  question  system  relevant  to  ones  field  of  specialization.  Consider  the 
questioning  systems  in  law,  medicine  or  science. 

In  Instructional  science,  questioning  can  be  useful  for  guiding  the  student 
toward  a  particular  logic  or  approach  to  the  subject  matter.  1  believe  that 
the  investigation  of  student  question  asking  will  provide  useful  information 
for  understanding  how  skills  develop  during  procedural  learning.  Evaluation  of 
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the  kinds  of  questions  asked  that  comprise  a  question  set  for  a  particular 
domain  should  provide  information  that  is  descriptive  of  the  knowledge  state, 
the  goals,  operators  and  methods  a  student  needs  at  a  given  point  in  time  to 
perform  some  task.  As  the  skill  behavior  becomes  more  practiced,  the  types  of 
questions  a  learner  asks  should  differ  as  a  function  of  the  knowledge  acquired. 
In  early  learning,  the  temporal  placement  of  the  question  in  relation  to  per¬ 
formance  may  reflect  the  nature  of  the  search  control  strategies  attempted 
during  the  task  as  different  problem  solving  methods  are  applied  to  the  problem 
space  for  the  task.  As  search  control  develops  into  more  precise  selection 
rules  for  the  domain,  this  temporal  placement  will  no  longer  have  significance 
and  the  question  may  disappear  altogether. 

This  paper  presents  a  model  of  an  analytical  methodology  for  interpreting 
questions  generated  in  procedural  skill  learning.  The  focus  is  on  knowledge 
representation  in  the  very  early  stages  of  procedural  skill  acquisition  in  a 
situational  learning  by  doing  paradigm.  The  first  section  introduces  the  con¬ 
cept  of  pedagogical  question  asking  and  understanding  the  function  of  these 
utterances  during  a  learning  task.  Next,  a  description  of  the  interpretive 
method  is  presented  to  illustrate  how  the  study  of  the  conceptual  analysis  of 
questions  can  help  psychologists  better  understand  how  knowledge  is  organized 
during  the  acquisition  process.  The  second  section  of  this  paper  discusses  the 
nature  of  the  domain  knowledge,  a  representational  framework  for  the  knowledge, 
a  popular  spreadsheet  command  language.  The  last  part  of  this  section  presents 
a  brief  overview  of  the  processes  and  strategies  employed  by  novices  in  proce¬ 
dural  skill  learning.  The  third  section  discusses  how  questions  might  function 
as  probes  in  memory  and  argues  for  the  utility  of  the  interpretation  model  for 
describing  knowledge  states  within  a  particular  representation  for  the  knowl¬ 
edge.  The  final  section  presents  empirical  evidence  and  discussion  of  the 
results  from  a  preliminary  investigation  employing  the  proposed  model  for  ques¬ 
tion  interpretation  in  a  procedural  learning  task. 

PEDAGOGICAL  QUESTIONS  AND  DISCOURSE 

Much  of  human  behavior  is  goal-oriented  in  language  communication.  Conver¬ 
sational  goals  can  serve  many  purposes:  getting  information,  imparting  informa¬ 
tion,  amusement,  or  simply  to  pass  the  time.  Questions  are  a  type  of  speech 
act  that  can  also  be  goal-based  (Belnap  &  Steel,  1976;  Hobbs  &  Robinson,  1979). 
In  natural  language  discourse,  questions  are  asked  for  many  reasons  and  can 
take  many  forms.  I  am  concerned  here  with  what  Harrah  (1973)  calls  pedagogical 
questions,  questions  that  can  provide  instructional  information  to  the 
questioner  when  answered. 

In  a  learning  situation,  students  will  ask  questions  to  obtain  the  informa¬ 
tion  they  need  in  order  to  understand  the  task.  Questions  are  asked  for  a 
variety  of  reasons  and  can  serve  many  purposes.  For  example,  consider  the 
questions  novice  users  of  computer  systems  might  ask  when  learning  a  new  skill 
on  the  computer.  Presumably  students  are  motivated  to  ask  a  question  because 
of  the  need  for  Instructional  information  regarding  system  or  task  use.  In 
procedural  skill  learning,  they  can  serve  to  gain  understanding  about  new  con¬ 
cepts,  new  procedures,  and  causal  relationships  in  the  domain.  From  a  psycho¬ 
logical  view,  the  study  of  these  pedagogical  questions  may  provide  information 
concerning  the  contents  and  organization  of  information  in  memory  during  skill 


acquisition.  Indeed,  Norman  (1979)  theorized  that  the  investigation  into  one's 
ability  to  ask  questions  can  be  useful  for  understanding  mental  strategies  used 
to  access  the  memory  system.  Educators  are  often  heard  to  lament  that  their 
students  never  ask  any  questions.  However,  this  is  not  always  true,  and  the 
educator  may  learn  a  great  deal  about  what  the  student  knows  from  the  study  of 
student-generated  questions.  In  fact,  educators  often  use  questions  as  an 
instructional  strategy  to  evaluate  learning  (Collins  &  Stevens,  1983;  Harrah, 
1973).  Encouraging  students  to  formulate  questions  about  what  they  are  learn¬ 
ing  can  develop  into  a  useful  strategy  for  understanding  the  subject  matter 
because  it  fosters  constructing  and  evaluating  hypotheses  about  the  meaning  of 
the  material  (Collins  &  Brown,  1986).  Miyake  and  Norman  (1979)  were  the  first 
to  state  that  the  ability  to  ask  questionsis  based  on  the  function  of  appro¬ 
priate  knowledge  structures  and  the  level  of  completeness  of  these  structures. 
However,  at  the  time  of  their  study,  there  existed  no  useful  method  with  which 
to  analyze  and  interpret  the  student-generated  questions.  There  currently 
exists  no  psychological  theory  of  question  asking  (Harrah,  1973;  Belnap  & 

Steel,  1976;  Norman,  1979),  therefore,  I  am  hopeful  that  the  results  from  this 
study  will  generate  useful  data  for  understanding  the  nature  and  purpose  of 
questions  in  a  learning  by  doing  paradigm  for  procedural  skills. 

Socratic  tutoring  is  based  on  the  use  of  questions  to  stimulate  students' 
reasoning  about  certain  concepts  (Collins,  1977).  Understanding  question  ask¬ 
ing  is  an  important  issue  for  creating  intelligent  tutors  wherein  both  the 
tutor  and  student  can  ask  questions  to  clarify  new  concepts,  student  perform¬ 
ance,  and  thereby  improve  both  learning  and  instruction.  Development  of  dia¬ 
logue  systems  in  computer  tutors  could  improve  instruction  by  allowing  the 
student  and  teacher/ tutor  to  engage  in  mixed-initiative  question  asking  for 
improving  understanding.  Recent  work  on  questions  has  focused  on  question 
answering  in  prose  comprehension  (Graesser,  1983)  and  in  artificial  intelli¬ 
gence  question-answering  and  intelligent  tutoring  systems  (Collins,  1977; 
Lehnert,  1978;  Burton  &  Brown,  1979;  Sammut  &  Banerji,  1986).  As  a  result  of 
some  of  this  work,  an  analytical  method  based  on  conceptual  categories  was 
developed  that  interpreted  questions  by  decomposing  the  question  particle  and 
remaining  proposition  of  the  utterance  within  the  context  of  the  query.  Little 
work  has  been  done  on  human  question  generation  and  its  implication  defining 
knowledge  representation  using  such  an  analytical  method.  None  has  been  done 
In  the  context  of  learning  procedural  skills. 

Analysis  of  questions  as  speech  acts  can  be  difficult  if  the  goal  is  not 
clear  from  the  utterance,  the  context  Is  unknown,  or  if  multiple  inferences  can 
be  Interpreted  (Hobbs  &  Robinson,  1979;  Belnap  &  Steel,  1976).  Here  I  address 
questions  that  are  not  uniquely  conversational  in  context  but  take  on  a  peda¬ 
gogical  motivation.  By  this  I  mean  that  the  context  of  the  questions  generated 
is  defined  by  the  learning  environment  and  nature  of  the  skill  behavior  re¬ 
quired  of  the  learner  performing  some  task.  Because  of  this  well-defined  con¬ 
text,  the  instructional  environment  will  necessarily  direct  the  goals  lying 
behind  any  speech  act.  During  learning  of  a  procedural  skill  such  as  command 
languages  the  first  two  interpretive  problems  of  goal  clarity  and  context  will 
be  obviated  by  the  nature  of  the  task.  This  point  will  be  discussed  in  more 
detail  in  the  section  on  command  language  learning  that  follows.  Because  the 
learners  are  naive  to  the  domain,  the  most  obvious  general  goal  for  their  ques¬ 
tions  will  be  to  locate  information  In  order  to  accomplish  and  understand  a 
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specific  task  or  knowledge  element  in  the  domain.  The  particular  time  within 
an  instructional  sequence  that  a  question  is  posed  will  further  define  its  goal 
based  on  what  the  student  knows  and  what  needs  to  be  learned  next  in  the  les¬ 
son.  Thus,  only  multiple  inferences  for  each  utterance  remains  a  potential 
|  concern  in  the  interpretation  process  for  understanding  the  meaning  of  peda- 

|  gogical  questions.  Still,  this  problem  alone  can  impose  significant  difficulty 

I  in  any  attempt  on  interpretation. 

Nevertheless,  borrowing  from  the  work  done  by  artificial  intelligence 
researchers,  I  feel  that  an  existing  methodological  approach  to  interpret 
utterances  using  a  conceptualization  scheme  to  classify  questions  will  help 
.  psychologists  understand  the  nature  of  question  asking  during  learning.  Compo- 

■  nents  of  a  methodology  that  can  reveal  the  conceptual  representation  of  a  query 

I  in  relation  to  the  contents  of  the  utterance  are  presented  below. 

;  ANALYTIC  METHOD  FOR  QUESTION  ASKING 

Conceptual  Question  Categories 

The  learner  who  attempts  to  accomplish  some  task  but  does  not  know  the 
appropriate  goal  structure,  actions,  or  operators  to  use  can  ask  a  question  to 
ellicit  the  required  information.  For  instance,  how  to  accomplish  a  goal  if 
the  correct  procedure  is  unknown,  soliciting  the  definition  of  certain  com¬ 
mands,  verifying  system  operators,  testing  hypotheses  regarding  proper  proce¬ 
dures,  or  clarification  of  solution  strategies.  It  seems  only  logical  that 
this  question,  when  analyzed  for  its  semantic  contents  and  meaning,  will  be 
linked  to  the  spot  in  the  accessed  knowledge  structure  that  this  missing  this 
information.  Research  investigating  a  conceptual  representation  for  questions 
i  in  question-answering  systems  has  attempted  to  resolve  some  of  the  issues 

related  to  understanding  natural  language  queries.  However,  several  problems 
remain:  understanding  the  meaning  of  a  question,  knowing  how  the  context 

affects  understanding,  selecting  appropriate  responses,  and  searching  memory 
for  the  response.  A  theory  of  question  answering  by  Lehnert  (1974)  has  identi¬ 
fied  thirteen  question  conceptual  categories  that  have  been  assessed  for  their 
1  validity  in  interpreting  certain  knowledge  concepts  from  question  utterances. 

!  They  are: 

1.  causal  antecedent 

2.  goal  orientation 

3.  enablement 

4.  causal  consequent 

3.  verification 

6.  disjunctive 

7.  procedural 

8.  concept  completion 
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9.  expec ta tional 

10.  judgement 

11.  quantification 

12.  feature  specification 

13.  requests 

She  has  developed  and  implemented  a  computer  program  that  can  generate  plausi¬ 
ble  answers  to  questions  formulated  from  these  categories.  This  is  akcom- 
pllshed  through  a  symbolic  procedure  for  specifying  the  world  knowledge 
structures  it  searches  to  obtain  the  answers.  Specific  rules  are  applied  to 
the  question  when  placing  it  into  the  appropriate  category.  For  example: 

Do  you  push  this  key  to  delete? 

(Verification  question  requiring  a  yes-no  response) 

The  questioner  is  asking  about  a  particular  operator  (a  key)  he  thinks  can  be 
used  to  delete,  but  seeks  clarification  through  asking  a  verification  question. 

The  result  of  applying  this  conceptual  categorization  methodology  renders 
the  utterance  language  independent  for  further  interpretation.  The  conceptual 
representation  provides  a  formalism  that  can  then  be  used  to  extract  the  mean¬ 
ing  from  the  utterance.  Lehnert  cautions  that  applying  this  method  does  not 
result  in  complete  understanding  of  the  question.  In  fact,  during  question 
Interpretation,  all  parts  of  the  proposition  must  be  analyzed  together. 

Graesser  (1985)  proposes  a  procedure  for  interpreting  questions  to  locate 
appropriate  answers  from  the  knowledge  base  or  memory.  He  incorporates 
Lehnert's  conceptual  category  procedure  for  interpreting  the  question 
particles  and  adds  another  procedure  to  interpret  the  remaining  utterance  or 
propositional  unit.  While  his  work  focuses  on  question  answering,  the  method¬ 
ology  he  employs  for  question  interpretation  is  appropriate  for  understanding 
the  nature  and  representation  of  questions  asked  and  for  tracing  the  path  of 
the  knowledge  element  sought  by  the  query  In  the  knowledge  structure. 

According  to  this  approach,  each  question  phrase  is  analyzed  by  identifying 
the  question  particle  and  placing  it  into  a  question  concept  category  to 
reflect  the  intent  of  the  question:  causal,  procedural,  feature  specification 
and  so  on.  The  propositional  unit  is  the  remainder  of  the  sentence  after  the 
question  particle  is  removed.  Next,  this  propositional  unit  is  divided  into 
two  components,  the  first  of  which,  the  statement  element,  is  categorized  into 
a  number  of  classification  types  to  reflect  the  context  or  main  idea  expressed 
by  the  proposition.  The  second  component  of  the  proposition  is  the  unknown 
element  of  the  question  or  what  Graesser  terms  a  knowledge  element.  The  knowl¬ 
edge  element  refers  to  the  kind  of  information  the  questioner  wants  to  have. 
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(<Question  TypeXContextXKnowledge  Node>) 

Lehnert's  categories  will  be  used  to  interpret  the  question  particles,  the 
first  element  in  the  Interpretation  model.  In  discourse,  propositions  express 
the  main  idea  the  communicator  wishes  to  Impart  in  the  utterance.  In  selecting 
a  classification  scheme  that  would  best  convey  the  ideas  central  to  question 
asking  in  procedural  skill  learning,  the  nature  of  the  domain,  command  lan¬ 
guages,  was  considered.  Given  this  domain,  I  have  chosen  three  classification 
types  that  I  feel  describe  the  context  of  the  propositional  unit  when  interact¬ 
ing  with  the  problem  space  presented  by  the  task:  goal,  action,  and  state. 

The  propositional  context  is  the  second  element  in  the  interpretation  model. 


A  cognitive  task  analysis  for  the  domain  based  on  the  GOMS  model  (Card, 
Moran,  &  Newell,  1983)  was  used  to  describe  the  structure  of  the  knowledge.  I 
will  present  a  brief  overview  of  procedural  skill  learning  and  a  theoretical 
framework  for  organizing  such  knowledge  after  it  develops  into  a  practiced 
cognitive  skill  in  the  following  section.  The  point  is  raised  here  to  estab¬ 
lish  the  rationale  for  selecting  the  three  classification  types  for  the  context 
in  the  propositional  unit  used  in  this  model.  Learners  begin  acquiring  some 
skill  using  problem  solving  strategies,  thus  we  can  think  of  the  propositional 
contexts  as  a  kind  of  problem  space  for  the  utterance.  Since  acquisition  of 
task  goals  are  important  to  procedural  skill  learning  as  are  acquiring  a  set  of 
correct  operators  necessary  to  perform  certain  acts  in  the  domain,  the  first 
two  main  ideas  I  expected  to  find  expressed  in  the  utterances  Included  those 
referring  to  task  goals  and  actions. 

Goal  problem  space  contexts  include  questions  about  task  goals  and 
subgoals.  Action  problem  space  contexts  include  questions  that  refer  to  opera¬ 
tors  or  operations  required  to  accomplish  a  specific  procedure  or  action.  The 
third  classification  type  selected  was  state  space  since  problem  solvers  are 
constantly  evaluating  states  as  they  search  for  the  correct  solution  path. 

State  problem  space  contexts  refer  to  any  question  about  a  state  transition  In 
the  system  that  results  after  some  operator  has  been  applied  to  some  existing 
or  current  state  or  a  particular  goal  has  been  obtained.  Since  there  may  be  a 
merging  of  two  kinds  of  knowledge  in  this  task,  the  domain  knowledge  of  the 
command  language  itself,  and  the  device  knowledge  for  the  computer  system, 
learners  may  have  questions  that  refer  to  a  particular  state  transition  that 
could  be  part  of  either  system.  Therefore,  the  state  problem  space  context  may 
refer  to  either  a  goal  state,  action  state  or  system  state  transformation. 

The  third  element  in  the  interpretation  model  is  the  knowledge  node.  Knowl¬ 
edge  sought  by  the  query  will  be  classified  into  general  schema-like  nodes  for 
the  domain.  These  nodes  should  reflect  the  broad  clusters  of  domain  knowledge 
that  students  are  formulating  and  accessing  in  order  to  accomplish  the  task. 
Similar  to  a  semantic  net,  these  nodes  will  have  little  organization  initially 
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and  serve  more  as  general  schema-1 Ike  representations  for  the  knowledge  until 
more  learning  and  practice  Impose  an  organizational  structure  on  them. 


To  interpret  the  semantic  content  of  the  propositional  unit  requires  some 
fundamental  lingulsltlc  interpretation.  However,  a  detailed  linguistic  study 
of  the  questions  generated  is  beyond  the  scope  of  this  paper.  1  will  focus  on 
simple  lexical  and  semantic  interpretation  and  refer  to  the  pragmatic  level  of 
discourse  in  relation  to  the  context  of  the  task.  Each  phrase  will  be 
decomposed  in  three  steps  into  a  representation  as  illustrated  here: 

Example  question: 

How  do  I  erase? 

1.  Linguisitic  Frame: 

(<question  par  ticleXproposi  tional  unit>) 

2.  Analytic  Frame: 

(  <questlon  typeXcontextXknowledge  node>) 

3.  Interpretive  Representation: 

(<ProceduralXGoal  SpaceXEd i  ting  node>) 

As  an  example,  in  the  context  of  the  task,  the  subject  had  considered  a  goal  of 
editing  whose  procedure  was  unknown  to  the  subject. 

Now  that  I  have  presented  the  concept  of  pedagogical  questions,  their  role 
in  learning,  and  an  interpretive  model  for  decomposing  the  questions,  1  will 
present  a  brief  overview  of  procedural  learning  in  command  languages. 

COMMAND  LANGUAGE  LEARNING 


Lotus  1-2-3 

Lotus  1-2-3  Is  a  command  language  for  creating  electronic  spreadsheets 
which  allows  the  user  to  input  data  in  a  two-dimensional  grid  space  displayed 
on  the  computer  screen.  A  variety  of  commands  and  functions  are  available  for 
manipulating  data.  Commands  are  called  using  a  menu  system  and  executed  after 
setting  various  parameters  by  pressing  the  'Enter'  key  or  carriage  return. 
Users  can  perform  sophisticated  numerical  calculations  by  formulating  and  exe¬ 
cuting  Lotus  specific  functions. 

Spreadsheets  can  be  saved  and  stored  in  computer  files,  used  to  create 
graphs  from  selected  data,  printed  as  hardcopy,  sorted  and  manipulated  as  a 
database,  and  even  programmed  to  perform  data  manipulation  in  user-specified 
ways  by  the  execution  of  Lotus  1-2-3  macro  statements.  The  domain  knowledge 
consists  of  a  series  of  specific  operators,  syntax  and  procedures  for  accom¬ 
plishing  these  tasks. 
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Because  command  languages  are  both  used  and  learned  on  computer  systems, 
the  interaction  of  the  device  model  for  the  computer  system  and  the  domain 
model  for  the  command  language  knowledge  the  student  employs  during  learning 
may  play  an  important  role  in  the  acquisition  process  (Kleras  &  Bovair,  1984). 
To  facilitate  human-computer  interaction  in  a  learning  situation,  a  common  rep¬ 
resentation  or  model  of  the  problem  or  task  is  desirable.  However,  in  order  to 
have  a  common  representation,  we  must  first  identify  the  representation  of  the 
task  in  light  of  the  user  and  the  system.  Command  languages  employ  a  com¬ 
mand-execute  cycle  (Moran,  1981)  in  their  application  which  differs  somewhat 
from  the  program-run-execute  cycle  of  traditional  programming  languages.  This 
difference  may  impact  on  how  the  procedures  are  learned  in  command  languages 
and  the  eventual  mental  model  for  the  knowledge.  In  order  to  describe  the 
mental  model  acquired  during  learning  a  skill  such  as  a  spreadsheet  command 
language,  we  must  first  define  the  component  parts  of  the  domain.  At  minimum, 
the  representation  of  initial  and  goal  states,  operators  for  performing  in  the 
domain,  and  the  condition-rules  required  to  complete  tasks  and  obtain  goals 
states  must  be  identified.  Once  completed,  we  can  begin  to  investigate  the 
role  these  components  take  in  the  formulation  of  the  mental  model  for  the  do¬ 
main  . 

Early  Learning:  Transition  to  GOMS 

The  learning  process  can  be  studied  using  an  incremental  model  to  describe 
the  knowledge  representation  at  various  stages  during  learning.  However,  the 
type  and  form  of  knowledge  acquired  and  used  at  these  various  stages  may 
require  different  representational  models.  The  student's  representation  may  or 
may  not  correspond  to  that  required  to  accomplish  the  task  (Kay  &  Black,  1984). 
This  is  particularly  the  case  for  novice  learners.  Card,  Moran,  and  Newell 
(1984)  propose  a  theoretical  framework,  GOMS  (goals-opera tors-rae thods-selec- 
tion),  for  organizing  the  hierarchical  goal  structure  for  complex  cognitive 
skills.  According  to  this  theory,  the  cognitive  structure  for  the  skilled 
behavior  consists  of  the  four  components  in  the  model.  However,  the  GOMS  model 
is  a  theoretical  construct  that  deals  only  with  well-learned  skills. 

Along  the  continuum  of  learning,  novices  rely  heavily  on  problem  solving 
behavior  for  performing  tasks  (Card,  Moran,  &  Newell,  1984)  until  an  adequate 
database  and  structure  for  the  knowledge  is  obtained.  Novices  begin  learning 
by  acquiring  and  building  a  declarative  knowledge  base  of  facts  and  features 
for  the  domain  (Anderson,  1982,  1985).  Because  the  structure  and  function  of 
the  knowledge  are  poorly  understood  at  this  early  period  in  the  learning  proc¬ 
ess,  novices  may  access  prior  knowledge  structures  of  similar  content  areas  to 
help  themselves  understand  and  process  the  information  until  a  sufficient  or¬ 
ganization  for  the  knowledge  Is  established  (Adelson,  1981;  Anderson,  1982; 

Kay  &  Black;  1984).  Continued  learning  refines  and  builds  on  the  initial 
knowledge  adding  appropriate  goals,  operators,  procedures,  and  rules  for  opti¬ 
mal  functioning  in  the  skill  area.  As  learning  continues,  skill  performance  In 
the  domain  changes  from  problem  solving  behavior  to  more  skilled  performance 
reflecting  instantiated  procedural iza tions  for  the  domain.  This  change  in 
behavior  should  be  reflected  in  the  changes  in  the  representation  of  the  stu¬ 
dent's  mental  model  as  well.  Novice  to  expert  differences  (Adelson,  1981; 
Soloway,  et  al,  1981;  Kay  &  Black,  198)  provide  further  evidence  for  these 
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structural  changes  in  knowledge  representation  as  a  function  of  learning  cou¬ 
pled  with  practice.  Kay  and  Black  (1984)  report  that  the  student  model  will 
take  on  goal,  state,  and  plan  changes  with  learning  that  will  allow  for  more 
expert  performance  as  expertise  develops.  However,  this  model  transition  is 
poorly  understood. 

Acquisition  of  Procedural  Skills 

Learning  involves  three  basic  stages:  encoding,  where  factual  or  declara¬ 
tive  knowledge  is  stored  in  memory;  proceduraliza tlon ,  where  facts  acquire 
function  and  are  transformed  into  procedures;  and  composition,  where  procedures 
gain  speed  in  execution  with  practice  (Anderson,  1985).  As  learning  pro¬ 
gresses,  the  knowledge  base  for  declarative  knowledge  increases.  With  little 
structure,  immediate,  'on-line'  interpretation  of  the  information  Is  required 
for  understanding  which  places  a  burden  on  working  memory  capacity  when  large 
amounts  of  new  information  are  being  processed.  Interpreting  declarative 
knowledge  is  inefficient  and  can  also  introduce  error  as  a  function  of 
retrieval  from  long  term  memory  when  dealing  with  a  poorly  structured  body  of 
knowledge.  Search  through  a  body  of  knowledge  takes  longer  when  organization 
and  structure  is  lacking.  Practice  of  the  skill  allows  repetition  of  proce¬ 
dures  and  certain  patterns  and  similarities  may  begin  to  emerge.  With  contin¬ 
ued  learning,  refining,  defining,  and  tuning  the  knowledge,  adequate  structure, 
organization,  and  procedural lza tions  are  formed.  Holyoak  (1984)  calls  this 
process  model  construction.  This  model  permits  more  efficient  search  in  memory 
that  results  in  better,  more  skilled  performance. 

Problem  Solving  Strategies 

When  students  begin  learning  a  new  domain  they  have  no  structure  for  the 
knowledge  and  they  employ  general  problem  solving  methods  to  find  solutions  to 
the  problems  they  encounter.  Newell  (1980)  calls  these  methods  weak  because  at 
the  very  early  stages  of  learning  when  the  knowledge  is  largely  unstructured, 
they  provide  little  power  for  finding  solutions.  Artificial  intelligence 
researchers  have  studied  may  of  these  weak  problem  solving  methods  in  order  to 
develop  simulation  models  for  the  learning  process  (Newell,  1980;  Neches, 
Langley,  &  Klahr,  1986).  Heuristic  search,  means-end  analysis,  and  operator 
subgoallng  are  but  a  few  of  these  methods  currently  identified.  Problem  solv¬ 
ing  is  a  common  learning  strategy  In  the  early  phases  of  learning,  however, 
this  strategy  can  also  occur  for  non-novices  when  the  instruction  lacks  spe¬ 
cific  flow  of  control  (Anderson,  1982)  or  when  the  task  as  problem  space  pre¬ 
sents  a  problem  to  the  person  faced  with  accomplishing  a  given  task  (Newell, 
1980).  Indeed,  even  experts  resort  to  problem  solving  when  faced  with  a  novel 
problem  in  a  given  domain. 

Anzal  and  Simon  (1979)  propose  a  theory  of  learning  by  doing  in  which  the 
learner  acquires  task  appropriate  problem  solving  strategies  through  the  dy¬ 
namic  process  of  attempting  and  applying  new  strategies  within  a  specific  task 
environment.  According  to  this  theory,  as  the  individual  applies  various 
weak-method  problem  solving  strategies,  a  correct  solution  path  can  eventually 
be  learned.  A  sequence  of  strategies  that  have  proven  successful  in  previous 
performances  will  be  applied  by  the  problem  solver  in  a  more  rapid  and  assured 
manner  in  subsequent,  similar  problem  spaces  or  task  subgoals.  An  analysis  of 
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the  learning  process,  such  as  that  reported  by  Anzal  and  Simon  (1979)  provides 
evidence  of  an  emerging  solution  strategy  based  on  prior  performance  within  the 
task  environment. 

From  this  discussion  one  can  easily  understand  that  problem  solving  methods 
will  be  the  predominantly  exhibited  behavior  for  novices  learning  new  skills. 
Soloway  (1986)  proposes  a  general  Incremental  problem  solving  model  for  learn¬ 
ing  programming  languages.  He  contends  that  this  Is  founded  on  the  fact  that 
expert  programmers  employ  this  strategy  and  that  by  encouraging  novices  to 
adopt  the  model,  their  learning  rate  should  improve.  Command  language  learning 
is  similar  to  programming  language  learning,  however,  Anderson  (1985)  suggests 
that  learning  by  Induction  appears  to  be  a  more  useful  strategy.  His  work  sug¬ 
gests  that  the  process  of  knowledge  compilation  will  transform  the  weak  problem 
solving  methods  into  domain-specific  productions  for  the  skill.  Perhaps  an 
incremental  problem  solving  model  could  provide  this  transformation  to  a  GOMS 
model  for  learned  skills  if  it  provided  for  the  compilation  of  procedures  at 
various  stages  during  learning.  Their  is  no  evidence,  however,  to  support  this 
claim. 


Skilled  Performance 

An  important  factor  for  the  success  of  the  knowledge  compilation  process  in 
transforming  the  weak  methods  into  viable  productions  that  lead  to  skilled 
performance  is  the  formation  of  accurate  goal  structures  for  the  knowledge. 

Goal  structures  are  defined  by  the  nature  of  the  task  and  the  procedures  re¬ 
quired  to  accomplish  the  goals.  This  factor  is  a  critical  instructional  issue 
and  education  based  on  presenting  these  structures  as  a  framework  for  the 
knowledge  can  help  improve  learning  by  making  the  goals  more  explicit.  Only 
through  learning  and  practice  of  the  skill  knowledge  is  an  appropriate  domain 
goal  structure  acquired.  This  goal  hierarchy  construct  is  a  central  component 
of  the  GOMS  model . 

Compiling  and  automatizing  skill  knowledge  are  discussed  in  detail  else¬ 
where  (Neves  &  Anderson,  1981,  Newell  &  Rosenblum,  1981)  and  the  reader  is 
encouraged  to  review  these  articles  for  additional  information  on  this  topic. 
The  point  to  be  made  is  that  all  novice  learners  adopt  problem  solving 
strategies  until  the  goal  structure  for  the  domain  is  learned.  Through  the 
learning  process  the  transformation  of  the  student's  problem-solving  model  to 
one  based  on  the  GOMS  framework  occurs  as  the  knowledge  becomes  procedure  1 1  zed 
in  service  to  the  goals  acquired  for  the  domain.  Thus,  skill  procedural iza tion 
in  a  well-defined,  goal-oriented  domain  allows  the  student  to  perform  more  ef¬ 
ficiently  as  function  is  added  to  the  form  of  the  knowledge. 

With  a  basic  understanding  of  pedagogical  questions,  an  interpretation 
model,  and  procedural  skill  learning,  we  are  now  ready  to  proceed  with  a  dis¬ 
cussion  of  the  functionality  of  questions  in  memory  and  how  their  interpreta¬ 
tion  can  describe  a  student's  knowledge  state. 


QUESTIONS  AS  PROBES  IN  MEMORY 


The  pedagogical  question  is  articulated  with  a  goal  of  obtaining  a  particu¬ 
lar  Instruction  or  knowledge  element  to  complete  understanding  of  a  given  skill 
component.  The  goals  of  these  questions  will  change  as  a  function  of  the  three 
interacting  representations  of  the  domain  knowledge  structure  during  learning 
(Newell  &  Simon,  1972):  the  student's  developing  internal  representation,  the 
external  representation  of  the  task  environment,  and  the  conceptual  representa¬ 
tion  used  for  instruction  during  learning.  As  the  student’s  representation  for 
the  knowledge  evolves,  changes  in  its  organization  will  depend  on  which  aspect 
of  the  domain  goal  structure  is  currently  being  learned  and  the  relevant  knowl¬ 
edge  the  student  needs  to  know  about  for  understanding  a  particular  concept, 
procedure  of  goal.  In  this  way,  the  student's  Interaction  with  the  domain 
knowledge  impacts  on  his  developing  representation  during  a  particular  instruc¬ 
tional  sequence.  The  student  actively  involved  in  accomplishing  a  particular 
task  in  a  situated  learning  by  doing  paradigm  will  respond  to  the  task  environ¬ 
ment  as  problem  space  and  how  it  is  perceived  in  developing  and  organizing 
knowledge  in  his  or  her  current  representation.  In  a  similar  manner,  the  cur¬ 
rent  instructional  sequence  and  which  particular  goals,  procedures  or  opera¬ 
tions  are  presentee  will  affect  what  aspect  of  the  knowledge  is  currently 
active.  Thus,  the  top  level  goals  of  the  questions  asked  will  change  depending 
on  which  level  a  student  is  currently  interacting  in  with  regard  to  the  domain 
knowledge  structure,  the  task  space,  and  the  instructional  sequence. 

A  presupposition  to  this  phenomenon  of  question  asking  is  that  the 
questioner  has  searched  memory  for  information,  found  It  was  not  available,  and 
evaluated  a  requirement  for  asking  a  question  to  obtain  what  is  needed  for 
understanding  or  to  perform  some  activity  in  the  task  environment.  In  order  to 
understand  how  pedagogical  questions  function  as  probes  in  the  memory  system  in 
a  learning  by  doing  paradigm,  three  components  requisite  to  an  inquiry  memory 
search  must  be  understood:  question  form,  level  of  search,  and  search  control 
s  tra  tegies . 

The  type  of  question  asked,  in  conjunction  with  its  contextual  meaning, 
will  determine  the  conceptual  'form'  of  the  question.  (Norman  (1979;  in  fact 
states  that  the  proper  form  for  the  query  is  necessary  before  a  search  in  mem¬ 
ory  can  be  performed.  In  order  to  further  define  the  question  form,  one  has  to 
understand  the  nature  of  the  domain  knowledge  that  will  be  accessed  by  the 
inquiry  memory  probe.  Neither  Lehnert's  nor  Grasser's  models,  which  dealt  with 
text  processing  as  the  domain  of  interest,  accounted  for  the  three  interacting 
representations  of  the  domain  of  the  knowledge  structure  when  discussing  memorv 
search. 

The  form  of  the  question  asked  will  necessarily  direct  the  query  to  one  of 
these  locations  in  memory  In  association  with  the  knowledge  node  and  the  par¬ 
ticular  element  accessed  within  a  node.  The  question  probe  may  solicit  informa¬ 
tion  that  deals  with  specific  operators  and  operations  for  accomplishing  some 
goal,  the  methods  with  which  to  carry  out  a  particular  operation  or  procedure 
or  even  the  initial  problem  solving  strategies  used  as  the  correct  methods  are 
acquired.  Questions  may  also  seek  knowledge  that  describes  the  concepts,  fea¬ 
tures,  properties,  and  selection  rules  for  a  particular  knowledge  element.  1  he 
form  will  define  the  path  of  the  query  as  it  links  itself  to  the  knowledge  node 


of  interest  and  leads  up  to  where  the  appropriate  answer  should  lie. 

Question  form  in  this  model  is  interpreted  by  coupling  the  question  type 
with  the  propositional  context.  For  example,  a  feature  specification  question 
about  a  specific  operator  property  will  direct  the  question  to  structural  in¬ 
formation  associated  with  the  knowledge  element  probed  within  a  particular 
node : 


"How  far  can  the  pointer  key  move  on  the  screen?" 

Here  the  keyboard  node  is  accessed  and  structural  information  about  spe¬ 
cific  operational  features  is  requested  about  a  particular  key.  On  the  other 
hand,  a  procedural  question  about  a  particular  operation  will  probe  for  func¬ 
tional  information  within  a  node: 

"How  can  I  center  this  title?" 

Here  the  data  entry  node  is  accessed  for  procedural  information  about  cen¬ 
tering  a  title  row.  As  learning  continues,  the  form  of  the  question  should 
take  on  changes  as  a  function  of  learning.  As  more  information  is  encoded  and 
stored  in  the  knowledge  nodes  in  the  developing  representation  for  the  knowl¬ 
edge,  the  student  will  have  more  information  on  which  to  draw  for  asking  ques¬ 
tions.  And  so  it  follows  that  the  organization  of  the  representation  will  take 
on  a  more  accurate  structure  as  domain  appropriate  goals  are  learned.  This 
will  allow  the  student  to  articulate  more  precise  questions  and  get  the  exact 
information  that  is  needed.  These  changes  in  form  will  in  turn  affect  the  path 
the  search  takes  in  memory  when  accessing  the  knowledge  representation  as  well 
as  the  level  of  the  search. 

The  level  of  search  in  the  memory  system  is  also  learning  dependent.  As 
the  student  learns,  more  information  is  acquired  and  a  domain  dependent  organi¬ 
zation  for  the  knowledge  structure  takes  form.  The  level  of  completeness  for 
the  structure  will  control  the  level  uf  search  when  a  question  is  formulated. 
Novices  in  any  domain  would  not  be  expected  to  ask  questions  about  skills  or 
information  they  don't  know  about.  Students  might  begin  by  asking  general 
questions  searching  across  the  nodes  in  memory  or  accessing  several  nodes 
rather  than  focusing  on  the  node  associated  with  a  particular  item  that  may  not 
yet  be  instantiated  within  the  representation  for  the  knowledge.  Thus,  at  the 
early  stages  of  learning,  a  breadth-first  search  of  memory  may  be  employed 
until  an  appropriate  organization  with  a  more  complete  knowledge  structure 
associated  with  the  student's  internal  representation  is  acquired.  The  novice 
student  will  attempt  to  locate  in  memory  what  it  is  he  needs  to  know  in  formu¬ 
lating  his  query;  so  to  for  the  more  practiced  student.  This  makes  intuitive 
sense  when  we  consider  what  is  known  about  knowledge  compilation,  spreading 
activation,  elaboration,  and  changes  in  the  knowledge  representation  during 
learning  (Anderson,  1982;  Card,  Moran  &  Newell,  1983;  Kay  S>  Black,  1984).  Con¬ 
tinued  practice  and  learning  in  the  domain  will  develop  more  complete  struc¬ 
tures  in  the  representation  and  the  level  of  search  when  questions  are  asked 
may  take  on  a  more  directed,  depth-first  approach.  Therefore,  different  ques¬ 
tion  forms  will  search  different  levels  of  memory  as  a  function  of  the  com¬ 
pleteness  of  the  structure.  This  completeness  of  structure  is  in  turn 
dependent  on  learning. 
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Other  factors  in  the  memory  retrieval  process  may  affect  the  level  of 
search  or  even  the  search  process  in  general.  How  memory  is  searched  is  a 
problem  researchers  continue  to  actively  investigate  and  it  is  a  process  that 
is  by  no  means  yet  completely  understood.  However,  such  factors  as  recency, 
decay,  and  forgetting  may  also  play  a  role  in  the  retrieval  process  (Neches, 
Langley,  &  Klahr,  1986).  Although  these  factors  are  of  paramount  importance  to 
memory  search,  they  are  not  the  focus  of  this  paper.  Nevertheless,  as  students 
begin  learning,  storing,  organizing,  and  retrieving  information  in  memory, 
their  question  formulation  process  may  be  affected  by  any  one  of  these  factors. 

Finally,  we  need  to  consider  the  search  control  mechanisms  that  apply  as 
questioner's  probe  memory  in  articulating  their  questions.  Question  function 
is  defined  by  relating  the  question  form  with  the  temporal  sequence  of  task 
space  interactions  (goal-action-state)  in  a  particular  learning  or  practice 
segment.  This  temporal/ sequen tial  component  will  serve  as  a  control  mechinism 
for  searching  the  knowledge  structure.  Search  control  can  be  defined  by  two 
factors  affecting  question  function:  system  response  or  feedback,  and  the 
instructional  sequence.  As  search  control  is  manipulated  by  performance  feed¬ 
back  and  instruction  (also  a  form  of  feedback),  question  function  will  shift 
and  ultimately  affect  the  strategies  used  to  probe  memory. 

Performance  feedback  will  direct  the  questioner  toward  a  certain  path  in 
memory  dependent  on  the  active  problem  space  in  which  he  is  interacting  by 
providing  the  student  with  information  to  evaluate  and  match  with  what  is  cur¬ 
rently  in  the  knowledge  structure  as  he  formulates  a  question  (Collins  &  Brown, 
1986).  In  much  the  same  way,  the  instructional  sequence  will  provide  the  stu¬ 
dent  with  situations  to  evaluate  with  what  is  currently  stored  in  memory.  The 
particular  time  within  an  instructional  sequence  that  a  question  is  posed  will 
define  search  control  based  on  the  completeness  of  the  knowledge  structure, 
what  is  known  about  the  goals,  operators,  methods,  and  selection  rules,  what 
needs  yet  to  be  learned,  and  the  strength  of  association  for  the  knowledge  or 
how  well  it  is  learned. 


As  search  control  is  altered  by  these  two  types  of  feedback,  so  to  is  the 
function  for  the  question.  Questions  may  serve  a  simple  information  retrieval 
function,  to  elucidate  information  about  a  causal  relationship,  specify  selec¬ 
tion  rules  for  a  particular  operation,  or  assess  an  hypothesis  about  some  task 
operator.  In  order  to  understand  more  fully  how  these  functions  affect  search 
control,  the  interpretation  model  must  define  the  mechanism  for  connecting  the 
question  probe  to  the  knowledge  node  accessed  with  the  student's  current  knowl¬ 
edge  representation. 

In  a  complete  question  system  there  is  the  query,  the  knowledge  base  or 
structure  accessed,  and  the  answer.  Here  I  am  only  concerned  with  interpreting 
questions  in  order  to  describe  the  questioner's  knowledge  state.  However,  to 
do  so,  I  must  address  the  entire  system  to  some  extent.  In  order  to  describe 
the  questioner's  knowledge  representation  through  interpretation  of  the  ques¬ 
tions  asked,  the  context  of  the  conceptual  domain  knowledge  structure  will  be 
used  to  define  the  representation.  This  structure  and  the  question  types  will 
be  used  to  define  the  input  relations  connecting  the  query  to  the  answer. 
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Figure  1  depicts  a  graphical  notation  for  the  types  of  probes  in  this 
model.  The  input  links,  representing  a  question’s  conceptual  category,  connect 
the  question  probe  to  the  knowledge  nodes  accessed.  Their  relations  are  de¬ 
fined  using  Lehnerts'  interpretation  rules.  The  context  classification  indi¬ 
cates  a  functional  distinction  that  further  defines  the  problem  space  for  each 
probe  as  it  traverses  the  knowledge  structure.  Thus  a  probe  continues  its 
search  in  either  a  goal,  action,  or  state  problem  space  associated  with  the 
knowledge  nodes  accessed  in  memory.  The  output  links  are  pointers  to  the  de¬ 
sired  Information  (the  answer).  Output  relations  define  functional,  structural 
or  process  components  associated  with  the  requested  information.  These  rela¬ 
tionships  might  Include  the  constraints  or  Implementation  process  associated 
with  legal  operators;  the  results  of  state  transformations  or  confirmation  of  a 
fact.  The  articulated  question  will  thus  probe  memory  by  pointing  to  a  par¬ 
ticular  knowledge  node  through  the  function  of  the  question  form.  What  the 
questioner  needs  to  locate,  but  does  not  yet  have  stored  in  memory  (or  cannot 
locate  because  of  retrieval  processing  constraints  imposed  by  the  instructional 
sequence  or  task  requirements),  is  the  output  link,  its  relational  definition, 
and  the  associated  information. 

IMPLICATIONS  FOR  KNOWLEDGE  REPRESENTATION 

How  Is  it  that  a  question  analysis  methodology  can  be  used  to  derive  a 
representation  for  the  knowledge?  A  complete  description  of  the  kinds  of  ques¬ 
tions  that  comprise  the  question  set  that  is  part  of  the  system  for  procedural 
learning  is  a  first  step  toward  understanding  the  role  of  pedagogical  questions 
in  the  development  of  a  knowledge  representation  for  the  domain.  Lehnert 
(1978)  maintains  that  In  order  to  answer  questions,  the  answerer,  whether  ma¬ 
chine  or  human,  must  retain  the  model  of  the  questioner's  knowledge  state. 
Evaluation  of  the  kinds  of  questions  asked  In  a  learning  by  doing  paradigm 
following  a  tutorial  session  should  provide  information  that  is  descriptive  of 
this  knowledge  state  model.  Changes  in  the  kinds  of  questions  asked  about  the 
operators  and  methods  a  learner  is  applying  to  solve  the  task  problem  should 
reflect  the  changes  in  his/her  knowledge  state  as  the  information  sought  is 
placed  in  the  student's  domain  model.  In  addition  to  interpreting  question 
types,  an  analysis  of  the  temporal  placement  of  the  question  in  the  protocols 
collected  should  reflect  the  nature  of  the  search  control  during  the  task  as 
different  problem  solving  methods  are  applied  to  the  problem.  Questions  asked 
at  the  boundaries  of  strategy  shifts  as  the  student  learns  the  appropriate 
solution  strategy  for  obtaining  a  task  goal  may  provide  information  for  under¬ 
standing  how  and  when  this  strategy  shift  process  occurs. 


I  believe  that  if  the  goal  hierarchy  for  the  domain  knowledge  is  well  de¬ 
fined  as  Is  the  case  for  learned  procedural  skills,  the  interpretations  of  the 
questions  will  point  to  specific  knowledge  nodes  in  the  domain  that  possess  the 


Figure  1  Links  and  Functional  Representation  for  the  Question  Probes 


information  the  questioner  is  seeking  in  relation  to  how  the  knowledge  is  being 
acquired.  Further,  the  sequence  of  the  questions  in  relation  to  the  dynamic 
problem  space  for  the  task  will  provide  additional  interpretive  information  for 
the  knowledge  state  changes  as  a  students'  internal  representation  develops. 

The  success  of  the  question  interpretation  method  relies  on  the  contextual 
constraint  of  the  domain  and  the  interaction  of  the  student  within  a  situated 
learning  by  doing  instructional  environment.  By  analyzing  the  questions  in  a 
well-defined  goal-based  hierarchical  context,  one  can  more  accurately  interpret 
the  utterances  and  be  more  specific  in  tracing  the  paths  the  query  takes 
through  the  knowledge  structure.  I  assume  a  hypothetical  organization  for  the 
domain  knowledge  based  on  concurrent  research  (Gray,  Mutter,  Swartz,  &  Psotka, 
1986)  that  is  attempting  to  define  novice  to  expert  differences  in  command 
language  learning  as  shown  in  Figure  2.  This  structure,  apart  from  the  editing 
goals,  was  used  as  part  of  the  instruction  in  the  experiment. 

An  analysis  of  the  kinds  of  questions  students  generate  while  learning  a 
skill  on  a  computer  may  provide  information  about  a  specific  knowledge  state 
that  can  be  mapped  unto  the  developing  representation  for  the  knowledge  as 
students  learn  the  concepts,  procedures,  and  skills  of  the  domain.  In  the 
learning  process  for  procedural  skills  such  as  command  languages,  student  ques¬ 
tions  should  indicate  what  particular  kinds  of  declarative  information,  opera¬ 
tors,  and  goals  they  need  in  order  to  perform  a  given  task  and  when  the 
information  is  needed.  This  information  should  indicate  the  mechanisms  by 
which  students  acquire  and  structure  the  newly  learned  knowledge. 

The  following  section  presents  some  preliminary  data  from  a  question  asking 
study  during  a  procedural  skill  learning  task. 

EXPERIMENT 


An  experiment  was  conducted  in  order  to  collect  baseline  question  asking 
data  in  a  procedural  skill  learning  task.  Novices  were  given  minimal  tutorial 
training  and  practice  prior  to  the  experiment  and  then  encouraged  to  solve  the 
task  problems  presented  to  them  in  a  learning  by  doing  paradigm.  The  proposed 


question  interpretation  model  was  used  to  analyze  the  data  from  verbal 
protocols.  Results  and  discussion  are  presented  below. 

METHODS 

Subjects.  Six  undergraduate  college  students  at  the  Catholic  University  of 
America  participated  in  this  study.  This  experiment  was  part  of  their  psychol¬ 
ogy  course  requirement. 

Materials.  The  command  language  under  investigation  was  LOTUS  1-2-3.  The 
task  involved  constructing  four  simple  spreadsheet  problems  on  a  computer. 
Cognitive  task  analysis  for  the  spreadsheets  was  undertaken  to  ensure  that  all 
four  problems  were  of  similar  difficulty  and  to  identify  the  goal-action  struc¬ 
ture  for  the  task.  Order  of  presentation  of  the  spreadsheets  was  randomized 
using  a  latin  square  design.  (See  Appendix  1) 
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Procedure.  The  experiment  was  a  wi thin-subjec ts  design  measuring  number  of 
questions  asked  per  spreadsheet.  The  trial  consisted  of  four  spreadsheet  re¬ 
construction  tasks.  The  trial  was  done  on  an  IBM-XT  equipped  with  a  hard  disk 
drive.  Subjects  were  administered  a  computer  skill  survey  after  informed  con¬ 
sent  was  obtained.  All  sessions  were  video-taped  and  audio-recorded  using  VHS 
video  recorder  equipment.  The  video  recorder  was  aimed  on  the  computer  screen 
and  positioned  unobtrusively  behind  the  subject.  A  keystroke  and  timing  pro¬ 
gram  were  used  to  capture  keyboard  behavior  and  time  on  task.  Each  subject  was 
given  fifteen  minutes  of  instruction  in  a  socra tic-like  tutoring  session  on  how 
to  move  around  the  spreadsheet,  enter  data,  and  create  simple  calculations  in 
LOTUS  1-2-3.  Five  minutes  of  practice  with  sample  exercises  was  conducted 
before  the  study  began.  The  sample  exercises  were  left  as  guides  during  the 
experiment.  No  constraints  were  made  on  completing  the  task.  Preliminary  ex¬ 
ercises  indicated  that  one  hour  would  be  enough  time  for  naive  learners  to  ac¬ 
complish  the  task.  The  experimenter  was  to  stop  the  subject  only  if  he/she 
remained  in  an  error  state  for  more  than  five  minutes.  Subjects  were  in¬ 
structed  to  reconstruct  the  four  spreadsheets  on  the  computer.  They  would  be 
helped  to  save  the  spreadsheet  and  begin  the  next  one  in  the  trial.  Subjects 
were  instructed  to  think  aloud  as  they  solved  these  problems  and  to  ask  ques¬ 
tions  aloud  when  they  weren't  sure  of  what  to  do  or  how  to  proceed.  Although 
the  questions  would  not  be  answered  until  after  the  session,  subjects  were 
encouraged  to  ask  questions  anyway  and  then  try  to  reconstruct  the  spreadsheets 
as  best  as  they  could.  After  the  session,  the  experimenter  would  provide  them 
with  any  information  about  LOTUS  1-2-3  that  they  wanted  to  know. 

RESULTS 

One  of  the  principal  goals  of  this  preliminary  study  was  to  begin  defining 
the  pedagogical  question  set  for  command  language  learning.  The  results  are 
presented  here  with  only  nonparame tr ic  analyses  of  the  questions  and  any  rela¬ 
tionships  between  them  and  the  knowledge  sought  by  the  query.  Once  a  basic 
question  set  is  defined  and  interpreted,  further  more  rigorous  analyses  of 
question  generation  will  be  performed  in  subsequent  research. 

Protocols  were  transcribed  with  questions  placed  on  a  separate  line,  but 
kept  within  the  text.  There  was  a  great  deal  of  variation  in  the  form  and 
content  of  the  utterances  and  using  the  context  of  the  protocols  was  essential 
for  interpreting  questions  in  cases  of  ambiguous  references,  ellipsis,  or 
anaphora.  Questions  were  analyzed  according  to  the  interpretation  methodology 
presented  above.  Individual  differences  were  noted  in  the  subjects  natural 
tendency  to  speak.  Some  were  simply  more  talkative  than  others  even  though  a 
practice  session  preceded  the  experiment  to  render  the  talk-aloud  method  more 
natural.  Also,  subjects  were  requested  to  ask  questions  and  this  may  have 
forced  an  unnatural  quality  into  the  experiment  that  might  not  be  truly  reflec¬ 
tive  of  a  typical  Instructional  setting  where  students  naturally  ask  questions. 
No  attempt  to  adjust  this  difference  was  made  in  this  pilot  study,  however, 
plans  for  future  research  will  do  so. 

Subjects  provided  a  total  of  166  questions.  Redundant  questions  were  sub¬ 
tracted  out  before  any  further  analysis  began  leaving  a  total  of  130  questions, 
an  average  of  21.7  questions  per  subject.  Figure  3  indicates  that  subjects 
asked  fewer  questions  as  time  went  on.  However,  this  overall  trend  was  not 
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evident  for  all  types  of  questions.  This  simple  factor  represents  an  overall 
tendency  for  asking  questions  during  learning.  As  novices  in  the  domain,  one 
expects  a  lot  of  questions  at  the  initial  stages  of  learning  and  this  finding 
appears  to  validate  that  fact. 

Seven  conceptual  categories  emerged  from  the  analysis  of  the  questions: 
procedural,  goal  orientation,  causal  antecedent,  causal  consequent,  conceptual 
completion,  feature  specification,  and  verification.  These  seven  conceptual 
categories  times  the  three  problem  space  context  categories  (goal,  action, 
state)  yielded  a  rather  large  21  category  matrix  of  question  types.  Table  1 
illustrates  the  categorical  breakdown  for  the  questions.  Three  categories  with 
the  highest  frequency  accounted  for  more  than  70%  of  the  questions  asked  during 
the  trial:  procedural  questions  (29%),  goal  orientation  questions  (22%),  and 
causal  antecedent  questions  (22%).  This  finding  is  not  too  surprising  as  one 
would  expect  these  sorts  of  questions  in  a  highly  procedural  domain  such  as 
LOTUS  1-2-3. 

From  the  total  number  of  questions  asked  (130),  three  knowledge  node  types 
accounted  for  64%  of  the  questions  asked,  data  entry  (35%),  editing  (15%),  and 
calculations  (14%).  To  determine  whether  there  was  any  relationship  between 
the  question  types  and  the  information  the  subjects  were  seeking  in  their  in¬ 
ternal  representations  for  the  knowledge  in  order  to  perform  the  task,  I  con¬ 
ducted  a  chi-square  test  on  the  data  matrix  for  the  most  frequently  occurring 
question  types  (procedural,  goal,  causal  antecedent)  and  knowledge  nodes  (data 
entry,  editing,  calculation).  Table  2  presents  total  frequencies  for  each 
category  in  the  matrix  and  chi-square  values  for  two  identified  relationships. 
Since  several  of  the  individual  cells  in  the  matrix  had  frequencies  less  than 
five  rendering  any  analysis  meaningless,  the  test  was  conducted  on  only  one  row 
and  one  column  of  data.  The  results  revealed  a  significant  relationship  be¬ 
tween  procedural  questions  and  the  three  knowledge  nodes,  X^(2,N=6)  = 
7.95,p<.05.  The  relationship  between  the  data  entry  knowledge  element  and  the 
three  question  types  approached  significance,  X^(2,N=6)  =  5.38,p<.05.  Although 
this  particular  test  cannot  specify  more  specific  associations  between  the 
data,  it  is  none  the  less  interesting  to  note  that  procedural  knowledge  is  most 
often  asked  about  both  in  question  type  and  knowledge  node. 

The  number  of  error-generated  subgoals  for  the  spreadsheets  was  calculated 
by  subtracting  the  total  number  of  task  subgoals  identified  from  the  total 
number  of  subgoals  each  subject  generated.  The  differences  in  total  subgoals 
per  spreadsheet  were  t(6)=2.03,p<.05,  for  spreadsheet  A;  t(6 )=1 . 32 , p< . 05,  for 
spreadsheet  B;  t(6)=0.90,  p<.05,  for  spreadsheet  C;  and  ( 6 )*0 . 73 , p< . 05 ,  for 
spreadsheet  D.  None  of  these  tests  were  significant  due  to  the  small  sample 
size.  However,  the  considerable  variation  of  subgoal  number  per  spreadsheet  is 
indicative  of  the  novice  problem  solving  methods  employed  by  the  subjects. 

Total  time  on  task  was  measured  for  each  spreadsheet  reconstructed  and  the 
mean  time  derived  across  subjects  to  reflect  the  learning  curve  for  the  task. 
Although  these  subjects  were  novice  learners,  the  theory  of  learning  by  doing 
(Anzai  &  Simon,  1979)  supports  the  notion  of  skill  acquisition  through  practice 


Table  1  Numerical  Breakdown  of  Questions  by  Conceptual  Category 
and  Context  Classification 
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Relationship  Between  Question  Probes 
and  Knowledge  Nodes  m  the  Internal  Representation 
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7  95.  p<  05  (procedural  row) 


5  38.  p<  05  (data  entry  column) 


f 

f 

I 

i 


while  solving  a  problem.  Because  four  spreadsheet  reconstruction  tasks  made  up  J 

a  trial,  I  had  anticipated  at  least  a  modest  learning  curve.  Results  using  , 

mean  times  are  presented  in  Figure  4.  Although  the  curve  is  not  dramatic,  it  1 

doe6  offer  some  support  for  general  learning  during  the  task.  However,  the  , 

problem  of  error-generated  subgoals  was  not  accounted  for  in  this  global  meas¬ 
ure.  Admittedly,  in  this  initial  study  subjects  were  not  provided  with  any 
tutorial  help  or  responses  to  their  questions  and  this  most  probably  contrib¬ 
uted  to  the  limited  overall  learning.  Also,  the  domain  involves  more  complex 
cognitive  skills  and  problem  spaces  than  in  the  Tower  of  Hanoi,  for  example, 
which  may  also  factor  into  the  overall  learning  curve  for  the  task. 

DISCUSSION 

Question  Interpretation 

Novice  learners  are  just  beginning  to  acquire  the  knowledge  necessary  to 
perform  the  target  skill.  They  tend  to  be  uncertain  about  what  procedures  to 
use  and  which  solution  strategies  will  work  best.  Therefore,  novices  in  gen¬ 
eral  are  more  inclined  to  ask  questions  about  these  skill  components  (Chi  & 

Glaser,  1980).  Any  reduction  in  question  asking  over  time  may  suggest  that 
novices  are  learning  the  correct  procedures  and  solution  strategies,  however,  1 
cannot  make  this  claim  based  on  the  data  here.  The  results  do  tend  to  confirm 
Chi  and  Glaser's  claim  for  general  high  frequency  question  asking  by  novices. 

Procedural  Questions.  These  question  types  were  the  most  frequently  asked 
question  type  in  this  experiment.  Procedural  questions  were  most  often  asked 
about  when  considering  a  goal  or  action  that  subjects  were  attempting  to  solve 
in  the  task  environment.  For  example: 

1.  "I  don't  know  how  to  do  it." 

(<ProceduralXAction  SpaceXData  Entry  Node>) 

2.  "I  don't  know  how  to  do  those  lines" 

(<ProceduralXAction  SpaceXData  Entry  Node>) 

3.  "I  can't  figure  out  how  to  move  the  mover  where  I  want" 

(<ProceduralXGoal  SpaceXCalcula tion  Node>) 

4.  "I'd  like  to  figure  out  how  to  move  one  space." 

(<ProceduralXGoal  SpaceXScreen  Movement  Node>) 

The  first  two  questions  are  seeking  appropriate  operators  to  accomplish  the 
procedure  under  consideration.  In  question  one  the  subject  had  already  se¬ 
lected  and  begun  work  on  a  task  subgoal,  entering  label  data  into  the  spread¬ 
sheet,  and  doesn't  know  how  to  set  the  CAP  LOCKS  key  for  capital  letters.  In 
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Spreadsheet 


Figure  4  Mean  Time  on  Task 
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question  two,  the  subject  asks  the  question  after  she  has  applied  an  operator 
that  has  proved  unsuccessful  for  obtaining  the  desired  subgoal  state.  As  a 
result,  her  question  refers  to  a  procedure  in  an  action  space  she  is  concerned 
about.  Compare  these  with  the  next  two  questions  that  are  posed  prior  to  con¬ 
sidering  other  subgoals  and  consequently  other  contextual  spaces.  The  verbs 
used  and  the  sequence  of  events  in  the  protocol  were  guides  for  discriminating 
between  space  contexts.  In  questions  three  and  four,  the  verb  'move'  is  used 
that  can  refer  to  an  operator  or  action.  However,  'when'  the  question  is  asked 
further  defines  the  context  classification.  These  questions  were  asked  before 
the  subject  began  another  subgoal,  so  the  temporal  quality  of  a  question  can 
change  the  meaning.  The  results  indicate  that  all  questions  asked  before  a 
subgoal  is  attempted  refer  to  goals.  When  questions  were  asked  after  a  subgoal 
has  been  approached  and  selected,  they  generally  referred  to  actions.  Proce¬ 
dural  questions  tended  to  increase  slightly  over  the  tasks  in  the  trial  which 
may  be  indicative  of  the  importance  of  knowing  about  correct  operators  for 
specific  actions  and  goals  in  the  early  stages  of  procedural  learning. 

Goal  Questions.  The  second  most  frequent  kind  of  question  asked  were  goal 
orientation  questions.  These  questions  were  asked  when  subjects  were  consider¬ 
ing  a  particular  task  subgoal  and  on  occasion  to  evaluate  some  system  state 
goal  they  encountered.  According  to  Lehnert,  goal  questions  are  typified  by 
their  inference  of  asking  for  a  'reason  for'  or  'possibility  of’  a  motive  or 
goal  behind  a  particular  action.  Her  analytic  approach  to  question  interpreta¬ 
tion  was  within  the  domain  of  text  processing,  and  this  renders  a  specific 
meaning  to  the  goal  orientation  conceptualization  category  she  used.  In  proce¬ 
dural  skill  acquisition,  however,  the  goals  and  subgoals  are  much  more  clearly 
defined  within  the  task  environment.  The  interpretation  here  was  facilitated 
by  the  cognitive  task  analysis  of  the  problem  space.  Each  spreadsheet  was 
broken  down  into  the  number  of  subgoals  required  to  complete  the  reconstruction 
task.  The  goal  decomposition  began  with  1)  global  task  goal  (reconstruct 
spreadsheet),  2)  task  subgoals  (the  number  of  columns  of  data  on  a  spreadsheet), 
and  3)  column  subgoals  (the  individual  rows  of  data  making  up  a  column).  Hence, 
any  question  asked  as  the  subjects  considered  or  referred  to  one  of  these 
subgoals  during  the  experiment  was  interpreted  as  a  goal  question.  Questions 
about  all  three  of  these  goal  categories  were  asked.  Goal  orientation  ques¬ 
tions  were  most  frequently  asked  in  the  first  task  trial  and  dropped  off  mark¬ 
edly  during  the  rest  of  the  tasks  in  the  trial.  For  example: 

1.  "I  wonder  if  I  can  'bump'  (erase)  that? 

(<Goal  Or  ien  ta  tionXGoal  SpaceXEd  i  ting  Node>) 

2.  “What  can  I  do?" 

(<Goal  Orien  ta  tionXGoal  SpaceXSpreadshee  t  Task  Node>) 

3.  "Now  where  am  I" 

(<  Goal  Or  ien  ta  tionXS  ta  te  SpaceXSpreadshee  t  Task  NodeX 


In  question  one  the  subject  Is  considering  an  editing  subgoal  and  in  number 
two  a  simple  task  subgoal  Is  being  considered.  Here,  the  subject  paused  in 
reconstructing  the  spreadsheet  and  considered  his  next  subgoal  decision.  Ques¬ 
tion  three  refers  to  goal  orientation  In  the  system  state  space.  Here  the 
subject  paused  during  the  task  and  scanned  the  spreadsheet  for  the  next  logical 
subgoal  to  consider. 

It  might  be  argued  that  both  of  these  last  questions  are  merely  rhetorical, 
but  further  analysis  of  the  pragmatics  of  these  utterances  within  the  context 
of  the  task  reveals  their  Intentional  Inferences.  Question  two  provides  an 
Indication  that  the  student  is  unsure  of  which  subgoal  to  logically  attempt 
next  within  the  problem  space.  Novices  tend  to  chunk  Information  within  a  task 
differently  than  experts  do  (Chi  &  Glaser,  1980).  Skilled  performers  know  the 
goal  structure  Inherent  in  the  problem  space  of  the  task  environment  and  there¬ 
fore  can  proceed  rapidly  in  the  selection  and  execution  of  task  subgoals.  This 
novice  subject  seemed  perplexed  by  the  task  at  this  point  and  was  seeking  guid¬ 
ance  through  a  question  In  selecting  the  next  subgoal. 

Question  three  carries  an  additional  component  of  spatial  orientation 
('where')  which  could  be  Indicative  of  either  a  memory  or  a  perceptual  problem 
In  scanning  the  spreadsheet  for  the  next  subgoal.  It  is  Indeed  difficult  to 
define  the  Inference  denoted  here,  however,  it  seems  reasonable  that  novices 
might  ' lose  track'  of  what  subgoal  they  should  consider  next  since  the  task 
goal  structure  is  not  well-formed  at  this  stage  of  learning.  This  'losing 
track  of'  phenomenon  could  be  a  combination  of  short  term  memory  overload 
and/or  perceptual  complexity  effects  on  spatial  orientation  for  an  unfamiliar 
task. 

Causal  Antecedent  Questions.  The  third  kind  of  question,  causal  antecedent 
questions,  were  asked  a6  frequently  as  were  goals  questions.  This  particular 
question  category  is  interesting  because  of  Its  Implications  for  hypothesis 
testing  and  reasoning  about  the  newly  acquired  skills.  Students  asking  causal 
antecedent  questions  are  asking  about  causal  relationships  between  certain  task 
states,  operators,  procedures,  goals,  and  the  like.  Questions  generating  hy¬ 
potheses  suggest  that  the  learner  is  beginning  to  create  some  sort  of  structure 
for  the  knowledge  (Miyake  &  Norman,  1979).  As  the  subjects  were  learning  how 
to  proceed  In  the  task,  what  operators  to  use,  the  correct  sequence  of  actions 
to  follow  and  so  on,  they  presumably  began  to  generate  and  test  hypotheses 
concerning  these  aspects  of  the  skill  knowledge.  When  the  system  response  was 
not  what  they  had  anticipated  based  on  their  hypothesis,  subjects  were  inclined 
to  ask  causal  antecedent  questions. 

For  example,  in  the  following  dialogue,  one  subject  was  trying  to  under¬ 
stand  how  pointer  keys  work  to  move  around  the  worksheet  space  on  the  computer. 

1.  "Do  these  things  here  (pointer  keys)  move  to  this  section?" 

(<Ver  If  lea  tionXAc  tlon  SpaceXScreen  Movement  Node>) 

2.  "Why  did  this  move  over  so  many  spaces?" 

(<Cau8al  An  teceden  tXS  ta  te  SpaceXScreen  Movement  Node>) 
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The  subject  is  attempting  to  understand  a  basic  concept  of  how  pointer  keys 
function.  Her  first  verification  question  suggests  that  she  already  thinks  the 
keys  she  is  referring  to  perform  the  operation  in  question.  As  a  novice,  she 
seeks  confirmation  of  her  hypothesis  through  a  simple  verification  question. 
Then  she  proceeds  to  apply  the  operator  to  test  its  function,  but  does  not  end 
up  in  the  anticipated  subgoal  state.  Thus,  question  two  seeks  causal  informa¬ 
tion  to  explain  the  current  unexpected  state  change.  All  subjects  were  given 
instruction  and  practice  on  pointer  keys  and  screen  movement,  however,  holding 
down  a  particular  pointer  key  continuously  causes  rapid  scrolling  in  the  direc¬ 
tion  of  the  pointer.  More  controlled  movement  occurs  when  the  key  is  depressed 
using  rapid,  serial  taps.  This  distinction  in  how  the  operator  works  is  an 
Important,  although  very  basic  one.  The  second  question  suggests  that  the 
subject  had  not  yet  learned  this  functional  quality  in  pointer  key  use.  In 
another  example, 

3.  "How  come  they  didn't  capitalize?” 

(<Causal  AntecedentXState  SpaceXData  Entry  Node>) 

a  subject  intended  to  set  the  CAPS  LOCK  key,  but  hit  a  neighboring  key  instead. 
In  evaluating  the  system  state  transformation,  he  asks  for  the  cause  of  the 
response.  Certainly  the  subject  expected  capitalization  and  doesn't  understand 
why  this  didn't  happen.  Of  course,  he  didn't  know  the  reason  was  because  of 
his  own  error.  Nevertheless,  this  type  of  question  is  useful  in  Illustrating 
how  he  Is  thinking  about  the  task.  In  the  protocol  he  tries  the  action  again 
and  then  reasons  that  he  must  not  have  had  the  CAP  LOCKS  key  depressed  earlier. 

Infrequent  Question  Types.  The  remainder  of  the  question  category  types 
accounted  for  only  312  of  the  total  questions  asked  and  thus  did  not  appear  to 
be  too  useful  to  students  for  acquiring  information  during  this  level  of  the 
learning  task.  Verification  questions  test  the  truth  of  a  statement;  they  re¬ 
quire  a  yes-no  response.  Novices  are  often  unsure  of  their  actions  and  how  to 
proceed  in  solving  a  problem,  hence  their  questions  may  simply  6eek  verifica¬ 
tion  of  some  fact  or  procedure  a  student  is  considering.  It  would  appear  that 
this  type  of  question  might  rise  In  frequency  as  more  experience  Is  gained  in 
the  early  stages  of  learning,  but  then  asymptote  when  a  specific  level  of 
learning  is  achieved.  When  students  become  more  confident  with  the  skill 
knowledge,  they  should  no  longer  seek  verification  through  this  type  of  ques¬ 
tion.  While  these  questions  were  not  overly  frequent  in  this  experiment  (12 
out  of  130)  when  compared  to  other  more  frequent  question  types,  they  did  vir¬ 
tually  drop  off  in  the  last  task  of  the  trial.  This  type  of  extinction  may 
indicate  that  at  a  certain  point  in  learning  they  are  no  longer  required  for 
confirming  knowledge.  This  question  type  was  the  most  difficult  to  Interpret 
in  the  pragmatic  sense  of  the  utterances.  In  a  few  cases,  although  on  the 
surface  the  question  appeared  to  be  one  of  verification,  a  specific  goal  or 
procedure  category  was  the  more  appropriate  Interpretation.  Much  as  the  Intent 
of  a  questioner  asking  "Do  you  have  the  time?"  is  really  interpreted  as  "What 
time  is  it?"  rather  than  a  yes-no  verification  of  the  fact,  questions  in  the 
protocols  such  as  ”16  there  a  way  to  cancel?"  are  really  asking  about  the  cor¬ 
rect  operator  and  are  interpreted  to  mean  "How  do  I  cancel?”,  a  procedural 
question  about  an  editing  goal  rather  than  simple  verification.  This  particu- 


lar  interpretation  classification  was  only  done  if  the  context  of  the  protocol 
and  keystroke  behavior  made  clear  a  more  specific  intention  for  the  question. 
Further  research  is  needed  in  order  to  test  the  notion  of  verification  question 
use  In  early  learning  stages. 

The  low  number  of  concept  completion  questions  is  surprising  (11  of  the  130 
total)  since  I  had  expected  novice  learners  to  ask  many  of  questions  of  this 
type.  Concept  completion  questions  are  like  ' fill-in-the-blank '  questions. 
Perhaps  in  later  learning  this  question  type  will  appear  more  frequently  as 
partially  learned  concepts  require  further  information  for  understanding  and 
refinement. 

Feature  specification  questions  refer  to  queries  soliciting  property  or 
attribute  Information  about  a  given  entity.  This  category  was  the  most  infre¬ 
quent  of  all.  One  might  think  that  because  novices  are  building  up  their  de¬ 
clarative  knowledge  base  for  the  domain,  that  they  would  be  interested  in 
learning  about  specific  features  of  task  entities.  Yet  at  these  early  stages 
of  learning,  the  novice  may  lack  the  required  knowledge  to  fine  tune  concepts 
through  this  type  of  question.  I  suspect  that  the  feature  specification  ques¬ 
tion  occurs  in  later  learning  stages  along  with  concept  completion  questions  to 
build  on  what  is  already  acquired  when  the  learner  has  a  basic  knowledge  struc¬ 
ture  intact  and  is  in  the  process  of  refining  and  structuring  the  knowledge 
into  a  more  meaningful  representation. 

Knowledge  Nodes:  Conceptual  Building  Blocks  in  the  Problem  Space 

In  describing  the  development  of  a  subject's  internal  representation  for 
the  task,  we  must  consider  two  factors:  system  feedback  as  a  result  of  his 
performance  in  a  given  problem  situation  (Newell  &  Simon,  1972)  and  the  in¬ 
struction  received  about  task  goals,  legal  rules,  and  concepts  that  apply  to 
the  situation.  The  process  by  which  one  develops  an  internal  representation 
can  include  one's  internal  thinking  responses  as  the  stimuli  and  perceived 
problems  are  dealt  with  (Newell,  1980;  Newell  &  Simon,  1972).  Articulated 
knowledge  about  a  particular  problem  in  the  task  environment  can  cause  this 
knowledge  to  become  available  for  solving  and  reasoning  about  the  problem 
(Collins  &  Brown,  1986). 

The  knowledge  nodes  that  subjects  articulated  through  their  questions  and 
verbal  protocol  statements  provide  evidence  that  they  are  encoding,  reasoning 
about,  and  accessing  these  areas  in  the  domain  as  they  begin  to  build  their 
Internal  representation  for  the  problem  space.  Learning  by  doing  promotes 
Individualized  exploration  in  this  space  within  certain  task  constraints.  The 
particular  knowledge  nodes  accessed  through  articulated  questions  identify 
different  kinds  of  knowledge  that  subjects  dealt  with  during  the  task  within 
this  learning  paradigm.  Figure  5  illustrates  how  these  question  probes  might 
function  in  the  early  stages  of  learning  as  students  build  a  representation  for 
the  knowledge. 

Nine  knowledge  nodes  emerged  from  the  questions.  These  elements  were  clas¬ 
sified  according  to  concept  nodes  that  reflect  the  highest  general  type  of 
knowledge  in  the  problem  space.  The  knowledge  nodes  identified  included:  1) 
data  type  *  this  conceptual  node  includes  declarative  information  sought  about 
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the  types  of  data  specific  to  LOTUS  1-2-3.  For  example,  label  (alphanumeric), 
value  (numeric),  and  formula  or  function  (calculation)  information.  2)  data 
entry  =  this  conceptual  node  includes  procedural,  operator,  and  methods  infor¬ 
mation  needed  to  enter  data  into  the  spreadsheet.  3)  editing  =  this  conceptual 
node  Includes  procedural,  operator,  and  methods  information  about  editing  data 
in  the  spreadsheet.  4)  spreadsheet  task  =  this  conceptual  node  refers  specific 
information  sought  about  the  task  goal.  5)  worksheet  features  =  this  concep¬ 
tual  node  refers  to  declarative  information  about  display  features  on  the 
screen  and  worksheet  parameters  (rows,  columns,  cell  addresses).  6)  system 

features  =  this  conceptual  node  refers  to  information  sought  about  computer 

system  behavior.  7)  screen  movement  *  this  conceptual  node  refers  to  both 
declarative  and  procedural  information  involved  with  moving  around  the 
worksheet  space.  8)  calculations  =  this  conceptual  node  includes  all  informa¬ 
tion  involved  with  creating  and  executing  calculations  on  numeric  data.  9) 

keyboard  =  this  conceptual  node  refers  to  declarative  information  about  key 

type  and  certain  specific  key  functions. 

The  kinds  of  knowledge  asked  about  reflects  the  nature  of  the  developing 
internal  knowledge  representation  and  what  the  student  regards  as  necessary  for 
accomplishing  the  task  goals.  Here  the  instructional  feedback  was  withheld 
during  the  experimental  task  and  only  system  feedback  affected  their  perform¬ 
ance  and  subsequently  the  form  and  function  of  the  questions.  However,  in 
classifying  the  different  kinds  of  knowledge  utilized  as  subjects  performed  the 
tasks,  it  appears  that  questions  probed  different  knowledge  structures:  either 
domain  or  device  knowledge,  and  within  domain  knowledge,  either  structural  or 
procedural  knowledge. 


Domain  Knowledj 

S  true  tural _ 

Da  ta  Type 
Keyboard 
Screen  Movement 
Spreadsheet  Task 


Procedural 

Da  ta  En  try 
Ed i ting 

Screen  Movement 


Device  Knowledf 


System  Features 
Keyboard 
Screen  Movement 


It  appears  that  some  of  the  knowledge  nodes  may  share  information  that  makes  up 
each  of  the  representations  for  the  knowledge. 

Subjects  were  provided  instruction  in  the  domain  knowledge  based  on  a  con¬ 
ceptual  representation  shown  in  part  in  Figure  2.  Certainly  data  entry  and 
calculations  were  two  concepts  presented  to  the  subjects  as  part  of  their 
raining.  Although  subjects  were  not  instructed  in  editing  knowledge,  they  did 
formulate  questions  that  accessed  this  node.  For  example,  the  high  proportion 
of  questions  asked  about  editing  seems  to  be  a  function  of  errors  generated  in 
the  task,  a  pheno  .enon  quite  natural  to  problem  solving  behavior  with  little 
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feedback  and  control  other  than  system  response.  The  reader  is  reminded  that 
the  instruction  for  this  experiment  was  confined  to  simple  data  types,  data 
entry  procedures,  screen  movement  procedures,  and  calculation  formation  and 
execution.  No  instruction  occurred  during  the  experimental  task.  No  instruc¬ 
tion  on  the  use  of  LOTUS  commands  was  provided,  and  given  the  nature  of  the 
reconstruction  task,  subjects  would  have  no  occasion  to  ask  questions  specific 
to  command  concepts  unless  this  aspect  of  the  domain  was  discovered  through  ap¬ 
plication  of  random  key  presses. 

Miyake  and  Norman  (1979)  stated  that  the  number  of  concepts  the  learners 
asked  about  implies  some  measure  of  association  between  the  student  and  the 
instructional  material.  Asking  about  a  particular  concept  implies  that  the 
questioner  knows  what  is  necessary  for  more  complete  understanding  of  the  con¬ 
cept.  Data  entry  questions  represented  the  most  frequent  knowledge  element 
asked  about.  Twenty-five  percent  of  the  questions  generated  about  data  entry 
knowledge  were  procedural,  goal  or  causal  antecedent  in  question  type.  This 
particular  knowledge  node  involves  the  most  basic  level  procedure  in  the  domain 
(See  the  TYPE-ENTER  METHOD,  P.  33).  It  is  the  first  procedure  acquired  in  the 
learning  process  and  essential  for  task  completion.  Data  entry  behavior  cannot 
be  avoided  in  the  task  environment.  The  proportionately  high  number  of  goal 
questions  for  data  entry  knowledge  is  Indicative  of  novice  behavior.  In  early 
learning,  even  the  most  fundamental  task  goals  and  subgoals  are  not  yet 
instantiated,  hence  more  of  this  type  question  is  asked.  It  also  follows  that 
a  high  number  of  procedural  questions  about  data  entry  knowledge  are  asked  to 
clarify  what  particular  operators  are  required  for  data  entry  in  various  situa¬ 
tions  . 

Editing  knowledge  was  the  second  category  most  frequently  asked  about. 

While  this  finding  may  appear  interesting  at  first  glance,  in  this  study  it 
only  reflects  the  subjects'  attempts  to  correct  the  many  errors  they  encoun¬ 
tered  during  the  trial.  The  reader  is  reminded  that  none  of  the  questions  were 
answered  and  that  subjects  had  to  revert  to  problem  solving  strategies  to  at¬ 
tempt  a  solution  as  best  they  could.  Since  no  instruction  was  given  about 
editing  commands,  one  would  expect  a  high  number  of  this  type  of  question  as 
subjects  expressed  a  desire  to  locate  the  operators  and  procedures  to  accom¬ 
plish  an  editing  subgoal.  However,  in  a  learning  environment  with  instruction¬ 
al  feedback  from  either  a  computer  tutor  or  teacher  (ie:  answers  to  the 
questions  asked),  the  need  to  ask  about  editing  knowledge  may  be  somewhat  lower 
than  was  observed  here.  Further  research  in  this  area  is  needed  for  validating 
this  claim.  Nevertheless,  these  results  indicate  the  importance  for  teaching 
editing  commands  and  operations  fairly  early  in  the  curriculum.  The  novice, 
who  will  be  making  a  high  number  of  errors  initially,  needs  to  have  an  under¬ 
standing  of  the  procedures  and  operators  necessary  to  correct  these  errors  in 
the  early  stages  of  learning.  Editing  knowledge,  although  not  a  primary  in¬ 
structional  objective  in  command  language  learning,  can  prove  useful  for  er¬ 
ror-prone  novices. 

Calculation  knowledge  was  asked  about  almost  as  frequently  as  was  editing. 
This  knowledge  element  was  more  task  specific  since  each  spreadsheet  task 
within  the  trial  had  at  least  one  calculation  as  a  subgoal.  This  particular 
subgoal  was  more  advanced  than  the  others,  for  the  most  part  basic  data  entry 


subgoals,  and  required  more  cognitive  effort  from  the  subjects.  Formulating 
calculations  requires  that  the  appropriate  row  or  column  of  data  is  identified, 
the  arithmatlc  operation  decided  upon,  and  the  appropriate  LOTUS  1-2-3  operator 
symbol  selected  to  perform  the  desired  calculation.  Two  forms  of  calculations, 
formulas  or  functions,  are  available  in  LOTUS  1-2-3  and  were  presented  to  the 
subjects  during  their  training.  Selection  of  one  type  of  calculation  over 
another  presupposses  the  decision  of  what  kind  of  calculation  is  needed  in  a 
particular  situation.  Often,  either  calculation  method  is  possible  and  the 
student  must  decide  which  one  to  use.  Most  of  the  questions  asked  about  the 
goals  and  procedures  involving  these  calculations.  The  goal  questions  offer 
evidence  for  this  calculation  type  decision  subjects  were  faced  with.  As  they 
gain  more  experience  within  the  domain,  this  type  of  goal  question  should  di¬ 
minish  in  frequency. 


Protocol  Analysis :  Prob 1 em  Solving  Strategies  Observed 


Protocols  were  analyzed  in  order  to  identify  the  most  common  problem-solv¬ 
ing  methods  the  subjects  employed.  Three  strategies  emerged  and  the  protocol 
statements  were  grouped  into  one  of  the  categories:  1)  means-end  analysis, 

2)  generate  and  test/operator  subgoaling,  and  3)’repeated  successful'  strategy. 
Further  discussion  of  these  strategies  is  available  in  other  sources  (Newell, 
1980;  Neches,  Langley,  &  Klahr,  1986).  The  third  strategy  is  named  by  the 
author  and  will  be  explained  below. 


A  problem  solver  using  a  means-end  strategy  involves  movement  in  a  forward 
direction  as  operators  are  selected  and  applied  to  move  the  problem  solver  from 
the  initial  state  to  the  goal  state.  When  a  means-end  strategy  proves  unsuc¬ 
cessful,  the  problem  solver  may  choose  to  apply  other  known  operators  or  to 
experiment  in  an  attempt  to  discover  the  correct  operators  required  for  achiev¬ 
ing  a  particular  subgoal  state.  This  sequence  of  generate  and  test  and/or 
opera  tor-subgoal ing  does  not  necessarily  move  forward,  but  moves  can  involve 
both  a  forward-backward,  backward-forward-backward  direction  as  different  op¬ 
erators  are  applied  or  new  subgoals  generated  in  which  the  operator  of  interest 
will  apply.  Application  of  a  'repeated  successful'  strategy  implies  that  the 
problem  solver  has  located  a  strategy  that  has  been  found  successful  for  a  par¬ 
ticular  class  of  subgoals.  Sweller  and  Levine  (1982)  call  'repeated  success¬ 
ful*  strategies  'history-cued'  since  it  implies  that  the  problem  solver  is 
remembering  a  strategy  based  on  the  individual's  performance  history.  This 
remembering  was  observed  in  this  study,  however,  Sweller  and  Levine  are  talking 
about  a  strategy  that  involves  considerably  more  learning  than  was  the  case 
here.  Certainly  with  more  experience  in  the  task,  I  would  expect  the  'repeated 
successful'  strategy  to  stabilize  and  shift  into  one  that  is  truly  his¬ 
tory-cued  . 


Problems  can  emerge  within  the  problem  space  as  a  function  of  the  interac¬ 
tion  between  the  task  goals  and  a  subject's  perceived  structure  of  the  environ¬ 
ment  (Newell,  1980;  Holyoak,  1984).  Weak  problem  solving  methods  will  be 
applied  to  these  problems,  declara tively  organized  in  a  novice's  representation 
prior  to  the  compilation  process,  to  help  interpret  their  implications  for 
performance  (Anderson,  1985).  It  was  clear  that  the  first  logical  strategy 
employed  by  all  subjects  was  means-end  analysis  by  selecting  a  goal  state  from 
the  spreadsheet  and  attempting  to  reach  that  state  by  applying  known  operators. 
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There  is  considerable  evidence  in  the  literature  to  support  a  novice's  use  of 
this  strategy  when  faced  with  a  new  problem  (Simon  &  Reed,  1976;  Newell,  1980; 
Neches,  Langley,  &  Klahr,  1986).  When  a  wrong  operator  was  applied  and  the 
goal  state  not  attained,  the  strategy  generally  reverted  to  a  sequence  of  gen¬ 
erate  and  test  and  opera tor-subgoaling  strategies  with  the  subjects  selecting 
other  operators  or  methods  for  accomplishing  the  subgoal.  If  a  solution  was 
eventually  obtained  using  one  of  these  strategies,  the  next  similar  subgoal  the 
subject  attempted  applied  the  same  strategy  based  on  the  previous  solution 
success.  When  a  solution  was  not  found  through  a  generate  and  test  strategy, 
operator  subgoaling  wa6  then  used  to  find  a  state  in  which  a  particular  opera¬ 
tor  would  work.  If  still  unsuccessful,  the  subject  would  either  abandon  the 
subgoal  or  after  five  minutes,  the  expiration  would  tell  the  subject  to  go  on. 
When  the  means-end  analysis  was  successful  and  a  succession  of  similar  subgoals 
remained,  for  example,  entering  a  column  of  labels,  the  strategy  became  more 
rapidly  applied  to  each  successive  subgoal.  The  terra  'repeated-successful'  is 
awkward,  but  conveys  the  meaning  behind  this  strategy.  The  student  has  found  a 
successful  solution  and  uses  it  repeatedly.  Evidence  of  this  type  of  behavior 
can  be  one  indicator  of  learning  as  search  control  time  reduces  for  a  particu¬ 
lar  subgoal  class  (Neches,  Langley,  &  Klahr,  1986)  using  this  strategy. 

Newell  (1980)  calls  this  'repeated-successful'  strategy  'planning'  to  imply 
an  abstraction  from  a  successful  means-end  analysis,  however,  this  terminology 
doesn't  seem  quite  appropriate  at  these  very  early  stages  of  learning.  Sub¬ 
jects  did  show  rudimentary  planning,  however,  since  the  domain  knowledge  was  so 
Impoverished  In  this  experiment,  I  would  hesitate  using  Newell's  term  for  this 
strategy  type.  Certainly  all  subjects  showed  evidence  of  planning  as  they 
attempted  different  solution  strategies,  but  their  'plans'  were  not  very  stable 
over  the  trial. 


Learning 


aestions  and  Strategy  Shifts 


Indicators  of  learning  include  composition  of  procedures  (Anderson,  1985), 
reduction  of  search  time  for  a  particular  procedure  (Neches,  Langley,  &  Klahr, 
1986),  and  organization  of  search  control  knowledge  Into  domain-appropriate 
methods  (Newell,  1980;  Card,  Moran,  &  Newell,  1983).  At  the  early  stages  of 
learning,  problem  solving  behavior  precedes  the  development  of  cognitive  skill. 
Practice  in  the  skill  transforms  this  problem  solving  behavior  Into  skilled 
cognitive  performance.  Here  I  present  results  of  a  more  fine-grained  analysis 
of  this  type  of  behavior  for  certain  types  of  basic  level  skills  the  student  of 
a  command  language  needs  to  acquire.  The  most  basic  level  procedure  identified 
in  LOTUS  1-2-3  Is  the  TYPE-ENTER  method  for  entering  label  or  numeric  data  into 
the  spreadsheet. 


TYPE-ENTER  METHOD 

SELECT-COLUMN/ROW 

MOVETO-COLUMN/ROW 

TYPE-DATA 

ENTER-DATA 


Observation  of  a  smooth,  rapid  execution  of  this  method  indicates  that  the 
individual  is  performing  a  cognitive  skill.  Subjects  quickly  learned  this 
method  and  went  from  means-end  analysis  to  a  'repeated  successful'  strategy  for 


this  particular  procedure  with  little  problem.  No  questions  were  needed  to 
clarify  understanding  about  this  procedure  after  the  instruction  and  practice 
session  prior  to  the  experiment. 


However,  each  spreadsheet  had  a  few  subgoals  which  were  more  difficult  than 
the  others.  These  more  advanced  level  subgoals  were  all  similar  in  that  they 
required  a  label-prefix  (a  special  symbol  in  LOTUS  1-2-3)  to  be  typed  in  prior 
to  the  contents  of  the  subgoal  in  question.  Analyzing  the  solution  part  at¬ 
tempts  and  strategy  shifts  for  these  subgoals  provided  more  detailed  insight 
into  the  changes  in  problem  solving  behavior  subjects  employed  as  they  au- 
tempted  to  solve  these  more  challenging  problems. 

Anderson  (1985)  reports  that  there  should  be  declarative  interference 
between  performing  a  skill  and  remembering  similar  knowledge  before 
proceduralization  of  the  knowledge.  This  appears  evident  in  the  protocols, 
especially  for  the  more  challanging  subgoals.  Subjects  began  by  applying  a 
means-end  analysis  to  solve  the  subgoal  problem.  In  the  sample  provided  below, 
entering  a  numeric-alphanumeric  data  type  codmbina tion ,  typing  a  special 
operator  before  the  data  type  is  required.  All  but  one  subject  forgot  the 
label-prefix  concept,  or  were  not  certain  of  how  to  use  it.  This  particular 
operator,  presented  in  the  tutoring  session  prior  to  the  experiment,  was  essen¬ 
tial  in  order  to  obtain  this  subgoal.  Although  there  isn't  enough  data  to 
represent  other  than  qualitatively,  subjects  generally  entered  a  means-end, 
generate  and  test  loop  in  an  attempt  to  locate  the  correct  solution  path  for 
this  subgoal  problem.  Questions  were  asked  at  the  boundaries  of  each  solution 
attempt  failure  as  current  states  were  evaluated.  Operator  subgoaling  was 
always  used  If  a  correct  solution  couldn't  be  located  with  a  generate  and  test 
strategy.  The  questions  were  often  followed  by  the  subject's  verbal  reasoning 
about  the  failed  strategy. 

After  the  Initial  typing  and  entering  "6  Month"  into  a  cell  address  (means- 
end  analysis  strategy)  and  finding  out  that  this  doesn't  work  unless  the  "6"  is 
preceded  by  a  label-prefix,  the  following  dialogue  occurred: 

Dialogue  S  tra  tegy 

failed  means-end 


Q1 

I  was  wondering  what  I  did? 

(<Causal  AntecedentXAction  SpaceXData  Entry  Node>) 

SI: 

OK,  I'll  try  again. 

genera  te 

and 

test 

S2: 

I  really  don't  understand  this. 

S3: 

One  more  time. 

genera  te 

and 

tes  t 

S4 : 

I'm  trying  to  think  of  another  way 
to  play  with  this. 

opera  tor- 

-subgoaling 

Q2:  Why  did  it  go  there? 


*-■ 


(<Goal  Or ien ta tionXAc tion  SpaceXData  Entry  Node>) 


At  each  strategy  shift,  the  subject  would  apply  various  different  operators  or 
sequences  of  operations  in  an  attempt  to  locate  the  correct  solution  strategy. 
This  subject  eventually  broke  up  the  subgoal  into  two  separate  subgoals  and 
entered  them  each  into  a  separate  cell  address  on  the  spreadsheet.  This  method 
was  not  the  correct  solution,  but  did  allow  the  subject  to  attain  the  subgoal 
state . 


Control  over  search  of  the  problem  space  depends  on  the  kind  of  knowledge 
one  has  Immediately  available  (Newell,  1980)  and  can  involve  applying  knowledge 
stored  in  memory.  If  a  goal  state  is  impeded  for  want  of  an  appropriate  action 
or  operator,  the  student  can  ask  a  question  as  a  memory  aid  to  gain  the  knowl¬ 
edge  needed,  and/or  can  apply  weak-method  problem  solving  strategies  to  attempt 
another  solution  path  in  the  problem  space.  Remembering  a  solution  strategy 
from  a  previous  occurrence  will  serve  to  facilitate  subsequent  problem  solving 
with  similar  subgoals.  In  another  case,  the  subject  remembered  a  similar 
situation  from  a  previous  spreadsheet  in  the  trial.  After  the  initial 
means-end  analysis  for  entering  a  formula,  the  following  dialogue  occurred: 


Dialogue 


Stra  tegy 
failed  means-end 


Ql: 


Q2: 


SI : 
Q3: 


How  come  it  didn't  go  in? 

(CCausal  AntecedentXAction  SpaceXCalcula  tion  Node>) 

Is  there  a  reason? 

( <Ver  if  lea  tionXAc  tion  SpaceXCalcula  tion  Node>) 

It  must  be  the  zeroes.  generate  and  test 


How  come  it  went  in  before  and 
it's  not  here? 


(<Causal  AntecedentXAction  SpaceXAction  Space>> 
<Calculation  Node>) 


S2: 


Skip  It.  I'm  going  to  get  rid 
of  that  zero. 


generate  and  test 


S3:  Well,  I'll  try  again. 


After  the  failed  means-end  strategy,  the  subject  asks  a  causal  antecedent 
question  Indicating  he  fully  expected  the  action  he  took  to  obtain  the  subgoal 
state.  The  next  question  Is  a  verification  type  with  the  verb  'reason'  implic¬ 
itly  indicating  that  the  subject  suspects  this  is  a  'reason  for'  the  action  not 
working.  The  causal  antecedent-verification  pair  of  questions  illustrates  some 
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hypothesis  testing  about  the  failed  strategy.  Even  though  this  subject  was  able 
to  retrieve  a  solution  strategy  from  memory  that  had  worked  on  a  previous  simi¬ 
lar  subgoal  (Q3:  entering  calculation  data),  unknowingly,  he  kept  hitting  a 
special  function  key  that  recalculated  the  formula  and  replaced  it  with  a  zero. 
Thus,  questions  can  be  used  by  students  to  retrieve  and  verify  pertinent  infor¬ 
mation  for  the  task.  However,  if  the  actual  behavior  exhibited  by  the  student 
is  errant,  the  solution  strategy  will  not  be  achieved. 

Because  the  learning  task  was  limited  to  controlled  instruction  and  feed¬ 
back,  no  dramatic,  stable  strategy  shifts  were  observed  over  subjects.  Simon 
and  Reed  (1976)  suggest  that  gradual  strategy  shifts  are  indicative  of  learning 
and  I  think  that  the  data  here  reflect  the  very  first  changes  that  occur  as 
students  learn  the  appropriate  rules  and  operations  that  make  up  the  best  solu¬ 
tion  strategy.  People  attempting  to  solve  problems,  and  faced  with  several 
choices  of  paths,  may  try  several  of  these  before  a  solution  is  found  (Simon  & 
Reed,  1976).  This  is  especially  true  for  novices  and  was  observed  when  sub¬ 
jects  first  applied  a  means-end  analysis  and  then  shifted  to  a  series  of  gener¬ 
ate  and  test  strategies  when  the  first  strategy  proved  unsuccessful.  Although 
the  goal  state  may  not  be  met  immediately,  trying  alternative  solution  paths 
can  serve  to  strengthen  more  low-level,  instantiated  skill  behavior  (here  the 
TYPE-ENTER  method),  and,  perhaps  lead  to  a  correct  solution.  Of  course  the 
learner  may  not  always  be  so  fortunate  and  risks  staying  lost  in  an  error 
state.  Because  the  domain  skill  in  this  study  is  more  complex  than  the  Tower 
of  Hanoi  problem  (Anzai  &  Simon,  1979)  and  other  traditional  problem  space 
domains,  this  strategy  transformation  process  should  naturally  take  longer  to 
develop.  More  instruction  and  practice  in  complex  skill  domains  are  required 
before  stable  strategy  shifts  can  be  observed  and  certainly  considerably  more 
experience  is  necessary  before  any  skilled  behavior  performance  can  be  expected 
to  be  observed. 

Another  indicator  of  learning  in  this  task  is  to  compare  the  number  of 
subgoals  expected  to  complete  the  task  with  the  number  of  subgoals  the  subjects 
actually  generated.  The  spreadsheets  used  in  this  study  were  very  simple  with  a 
spreadsheet  title,  one  column  of  label  data  (names  of  items),  one  or  more  col¬ 
umns  of  number  data,  and  one  or  two  calculations  to  be  performed  on  the  number 
data.  Goal  decomposition  resulted  in  a  total  number  of  subgoals  for  each 
spreadsheet.  Although  the  total  number  might  vary  between  the  spreadsheets, 
the  types  of  subgoals  were  common  within  spreadsheets.  Most  of  the  errors  were 
committed  because  of  simple  typographic  errors,  and  omitted  portions  of  the 
spreadsheet.  The  omissions  were  presumably  due  to  how  the  subjects  visually 
'chunked'  the  spreadsheets,  however,  this  paper  will  not  address  chunking  phe¬ 
nomena.  Other  errors  were  committed  because  the  required  operators  were  simply 
not  available  in  the  subjects'  representation  for  the  knowledge.  Some  of  these 
subgoals  were  simply  not  attempted  by  subjects  with  them  declaring,  "I  don't 
know  how  to  do  that  so  I'll  go  on."  Further  work  is  planned  to  discuss  errors 
in  detail.  It  is  mentioned  here  to  illustrate  its  impact  on  novices'  overall 
learning  rate.  This  data  suggests  that  without  positive  reinforcement  or 
remediation,  error-generated  subgoals  can  grow  unchecked  which  can  impose 
constraints  on  effective  learning  and  foster  misconceptions  in  understanding 
the  skill  knowledge.  Although  subjects  could  often  find  the  correct  solution 
strategy  through  a  successful  sequence  of  means-end  analysis/generate  and  test 


strategies,  most  often  the  correct  solution  path  was  not  located  until  after 
several  unsuccessful  paths  were  tried.  Clearly,  this  unguided  practice  is  not 
an  efficient  learning  strategy. 

CONCLUSION 

The  results  of  this  study  indicate  that  a  specific  set  of  questions  rele¬ 
vant  to  the  domain  of  procedural  skill  learning  does  indeed  exist.  This  ques¬ 
tion  set  comprises  a  series  of  question  types  for  soliciting  certain  kinds  of 
knowledge  in  an  instructional  setting  that  appear  to  play  specific  roles  in  the 
learning  process.  Problem  solvers  attempting  to  learn  the  structure  of  a  given 
task  can  combine  question  asking  with  evaluation  of  individual  performance 
behavior  from  previous  similar  problem  solving  situations  to  organize  and  un¬ 
derstand  task  knowledge  in  a  learning  by  doing  paradigm. 

Pedagogical  questions  function  in  general  to  identify  a  problem  the  student 
has  with  understanding  a  particular  concept  or  procedure  in  the  domain.  Ques¬ 
tions  can  also  serve  as  memory  cues  when  all  of  the  information  can't  be  re¬ 
trieved  when  it  is  needed.  To  articulate  a  question,  the  student  must  be  able 
to  express  what  it  is  he  or  she  needs  to  know.  One  basic  assumption  for  the 
ability  to  do  this  is  that  the  student  has  already  acquired  a  significant  body 
of  knowledge. 

A  tripartite  cluster  of  question  types,  procedural,  goal,  and  causal  ante¬ 
cedent,  appears  to  function  as  the  primary  method  for  acquiring  and  understand¬ 
ing  prerequisite  task  knowledge  In  early  learning.  Procedural  questions  were 
by  far  the  most  frequent  question  type  generated  in  this  study  suggesting  their 
importance  for  understanding  how  legal  operators,  actions,  and  procedures  are 
used  at  the  very  early  stages  of  learning.  The  knowledge  element  accessed  in 
these  procedural  questions  defines  the  higher  level  nodes  in  the  representation 
that  a  particular  question  refers  to.  I  suggest  that  procedural  questions  will 
retain  a  high  frequency  until  basic  task  procedures  become  instantiated  in  the 
student's  representation  Jor  the  domain.  A  drop  in  frequency  for  this  question 
type  should  identify  the  beginning  of  the  proceduraliza tion  stage  for  the  par¬ 
ticular  skill  in  question.  Goal  questions  appeared  to  drop  off  somewhat  dra¬ 
matically  after  the  first  encounter  with  the  task.  These  questions  seem  to  be 
most  useful  to  novices  who  are  completely  unfamiliar  with  a  task.  Once  a  basic 
orientation  to  the  task  as  problem  space  Is  attained,  goal  questions  no  longer 
serve  a  pedagogical  purpose.  Goal  questions  were  observed  to  refer  to  overall 
task  goals  and  did  not  function  in  acquiring  knowledge  about  the  domain's  or¬ 
ganizational  goal  structure.  This  question  type  occurs  only  at  the  beginning 
of  learning  as  a  task  orientor  for  novices.  I  suspect  the  same  question  asking 
behavior  will  be  true  for  experts  when  faced  with  a  difficult  or  novel  problem. 
Novices  use  verification  questions  alone  to  confirm  what  they  think  they  know 
but  are  unsure  of,  and  in  conjunction  with  causal  antecedent  questions  serving 
to  refine  reasoning  about  relationships  between  concepts  and  to  test  hypotheses 
about  skill  behavior.  Miyake  and  Norman  (1979)  suggested  that  students  who  are 
capable  of  generating  and  testing  hypotheses  are  in  the  process  of  actively 
constructing  some  type  of  organizational  structure  for  the  knowledge.  The  use 
of  a  verification-causal  antecedent  questioning  sequence  offers  evidence  in 
support  of  this  claim.  Use  of  this  sequence  of  questioning  should  appear  more 
frequently  when  students  are  structuring  a  particular  knowledge  element.  I 
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think  this  type  of  questioning  will  become  more  prevalent  in  the  intermediate 
stages  of  learning  as  students  begin  to  procedural ize  and  organize  the  declara¬ 
tive  knowledge  obtained  at  the  the  beginning  level  of  instruction. 

Other  more  infrequently  asked  question  types  may  simply  not  be  needed  by 
novices.  Feature  specification  and  concept  completion  questions  may  only  have 
pedagogical  usefulness  to  the  student  after  a  sufficient  body  of  knowledge  has 
been  acquired  and  a  basic  organizational  structure  formulated.  The  appearance 
of  these  question  types  in  conjunction  with  the  knowledge  element  the  student 
wants  to  know  about  should  reflect  when  in  learning  a  particular  area  of  knowl¬ 
edge  requires  refinement  and  tuning. 

Results  from  this  study  may  also  have  implications  for  a  theoretical  basis 
of  the  psychology  of  question  asking  in  a  procedural  skill  domain.  Belnap  and 
Steel  (1976)  and  Harrah  (1973)  argue  for  the  need  to  analyze  the  logic  of  ques¬ 
tions.  Here  I  present  some  preliminary  data  that  begins  to  evaluate  the  psy¬ 
chological  logic  of  a  set  of  pedagogical  questions  used  to  acquire  and  organize 
a  particular  kind  of  skill  knowledge.  As  a  theory  of  pedagogical  question 
asking  develops,  we  can  begin  to  understand  the  nature  of  questions,  their 
purpose  in  instructional  environments,  and  their  implications  for  defining 
knowledge  states.  The  classification  and  analysis  of  questions  asked  through¬ 
out  various  stages  of  learning  should  facilitate  our  understanding  of  how  and 
what  kind  of  questions  are  used  at  different  levels  of  learning.  A  more  pre¬ 
cise  identification  of  the  specific  kinds  of  knowledge  a  student  needs  during 
the  various  phases  of  the  acquisition  process  should  lead  to  a  more  rapid  tran¬ 
sition  from  a  problem  solving  mode  to  practiced  cognitive  skill  performance. 

Understanding  question  asking  and  its  role  for  defining  student  knowledge 
states  is  important  for  developing  more  effective  instructional  strategies.  One 
effort  in  this  regard  is  the  current  research  in  intelligent  tutoring  systems. 
For  the  tutoring  system  to  be  most  effective,  it  must  know  the  student's  knowl¬ 
edge  state  so  that  the  tutor  can  monitor  and  deliver  appropriate  instruction. 

A  formalism  for  representing  pedagogical  questions  might  help  the  representa¬ 
tional  problems  surrounding  s tudent- tu tor  dialogues  in  these  systems.  One  ex¬ 
ample  is  a  system,  MARVIN  (Sammut  &  Banerji,  1986),  that  understands  and 
answers  verification  questions  for  simple  concept  learning.  Tutoring  systems 
need  to  know  how  much  and  what  kind  of  information  the  student  knows  before  an 
appropriate  response  can  be  given.  If  the  tutoring  system  can  interpret  the 
knowledge  state  from  the  types  of  questions  a  student  asks,  than  an  appropriate 
answer  is  possible.  However,  much  work  needs  to  be  done  in  this  area  before 
robust  dialogue  systems  in  intelligent  tutors  become  a  reality. 

SUMMARY 

The  research  described  in  this  paper  has  been  undertaken  in  order  to  inves¬ 
tigate  the  role  of  question  asking  during  procedural  skill  acquisition.  The 
approach  taken  to  interpret  the  meaning  of  the  questions  has  attempted  to 
analyze  the  questioner's  discourse  goals  through  a  conceptual  categorization 
method  within  a  specific  context.  From  the  preliminary  data  analysis,  one  can 
see  evidence  of  an  emerging  question  set  Inherent  to  the  domain.  Interpreta¬ 
tion  of  the  type  of  questions  asked  as  well  as  their  placement  within  the 


Instructional  context  appears  to  provide  information  that  is  useful  for  dg- 
scribing  a  particular  knowledge  state.  Understanding  the  function  of  questions 
during  learning  and  how  to  interpret  their  meaning  may  be  useful  for  the  devel¬ 
opment  of  dialogue  systems  in  intelligent  tutoring  systems.  Continued  research 
is  necessary  in  order  to  elucidate  further  the  question  system  in  learning,  its 
implication  for  knowledge  representation  and  how  understanding  questions  can 
improve  instruction  as  well  as  skill  acquisition. 
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APPENDIX  A.  SPREADSHEET  USED  AS  STIMULUS  MATERIAL 
WITH  SUBGOAL  DECOMPOSITION 


AVERAGE  UTILITY 


Utility 

6  Month 

Month 

Cost 

Average 

January  1982 

6500 

February 

6700 

March 

6400 

April 

5700 

May 

4900 

June 

4100 

5770 

July 

4300 

5350 

August 

4500 

4980 

September 

4200 

4610 

October 

4700 

4450 

November 

5100 

4480 

December 

5800 

4760 

Averages 

•  * 

•  * 

••(Create  a  formula  or  function  to  average  numbers  in  these  columns) 


TOTAL  SUBGOALS  =  47 
Alphanumeric  =  17 
Numeric  =  19 
Alphanum/num  combo  =  2 
Symbols  =  7 
Caludations  =  2 


